PowerPoint Presentation

Introducing the Amazon S3

Glacier Instant Retrieval

storage class

Jessie Felix

S T G 2 1 4

Product Manager, Amazon S3

AWS

Andrew Kutsy

Product Manager, Amazon S3

AWS

Srinivas Bobba

Principal Product Manager

Splunk

Patterns of cost optimization

Using Amazon S3 storage classes

Introducing the S3 Glacier Instant Retrieval storage class

Introducing the S3 Intelligent-Tiering Archive Instant Access tier

How Splunk optimizes cost with Amazon S3

Agenda

How are customers using S3?

Media master files

DNA

sequences

Mobile sync and storage

ML training data

Media assets

Internet of Things (IoT)

sensor data

Data lakes

Durable

backups

Digital record

preservation

Seismic and

reservoir

simulation data

Home-

recording video

Meteorological and

environmental research

Origin storage

for CDN

Financial

transaction

records

Website

hosting

Surveillance

video/closed-

circuit television

Medical images

and records

Autonomous

vehicle data

Oil and gas topography

Customer

call-center

records

Geospatial or lunar imagery

Compliance

records

Pharmaceutical

study data

User-generated content

Log files

Analytics

Amazon

S3

Operating at scale in Amazon S3

Wide range of

cost optimization

capabilities

Industry-leading performance

Broadest data movement

and hybrid cloud

storage options

Amazon S3

Industry-leading scalability,

availability, and durability

AZ

Benefits of multiple AZs in each AWS Region

Amazon S3

Redefining cost

optimization with S3

Introducing S3 Glacier Instant Retrieval

Designed for

99.999999999%

durability

$0.004 per

GB-month

Milliseconds

access

Designed for

99.9%

availability

Lowest cost storage with milliseconds

retrieval for rarely accessed data

Introducing Archive Instant Access tier in

S3 Intelligent-Tiering

Frequent

Access tier

Infrequent

Access tier

Archive

Instant

Access tier

+30 days +60 days

New

Automatically delivers

up to 68% in storage

cost savings

Patterns of cost optimization

1

Data with known or predictable access patterns

2

Data with unknown or changing access patterns

Patterns of cost optimization

Data with known or predictable access patterns

Data with unknown or changing access patterns

2

1

Patterns of cost optimization

1

Data with known or predictable access patterns

Data with unknown or changing access patterns

2

Data with unknown access patterns

Data Lakes

Data Analytics Long Tail Data

Content Sharing

A C R O S S E V E R Y I N D U S T R Y

Data with known access patterns

Medical Records User Generated

Content

Learning Resources Backups

Why we built S3 Glacier Instant Retrieval

Archive data stored for

several years

Very low rate of access Immediately accessible

when requested

The lowest cost archive storage with

milliseconds retrieval

Using Amazon S3

storage classes

Amazon S3 storage classes

O P T I M I Z E Y O U R S T O R A G E C O S T B Y U T I L I Z I N G A L L A M A Z O N S 3 S T O R A G E C L A S S E S

S3 Standard

(2006)

S3 Glacier

(2012)

S3 Standard-IA

(2015)

S3 One Zone-IA

(2018)

S3 Intelligent-

Tiering

(2018)

Accelerating innovation

2006

2021

S3 Glacier

Deep Archive

(2019)

S3 Outposts

(2020)

2006 2021

Decreasing storage prices

S3 Glacier

Instant Retrieval

(2021)

S3 Intelligent-

Tiering, Archive Instant

Access

(2021)

New

Your choice of Amazon S3 storage classes

AWS Region  3 Availability Zones

S3 Standard

S3 Standard-IA

S3 One

Zone-IA

S3 Intelligent-

Tiering

S3 Glacier

Deep Archive

Re-creatable,

infrequently accessed

data

S3 Outposts

On-premises data

AWS AZ AWS Outposts

Frequently

accessed data

• Milliseconds

access

• No retrieval

charge

Changing access

patterns

• Milliseconds

access

• No retrieval

charge

• Archive Instant

Access tier

Infrequently

accessed data

• Milliseconds

access

• Per-GB retrieval

charge

Archive data

Long term archive

data

S3 Glacier

Instant

Retrieval

Rarely

accessed data

New

S3 Glacier

Flexible Retrieval

(formerly S3 Glacier)

• Milliseconds

access

• Per-GB retrieval

charge

• Retrieval options

from minutes to

hours

• Free bulk

retrievals

• Retrieval in hours

• Milliseconds access

• Per-GB retrieval

charge

• Milliseconds access

New

Your choice of Amazon S3 storage classes

S3 Standard

S3 Standard-IA

S3 One

Zone-IA

S3 Intelligent-

Tiering

S3 Glacier

Deep Archive

Re-creatable,

infrequently accessed

data

S3 Outposts

On-premises data

Frequently

accessed data

• Milliseconds

access

• No retrieval

charge

Changing access

patterns

• Milliseconds

access

• No retrieval

charge

• Archive Instant

Access tier

Infrequently

accessed data

• Milliseconds

access

• Per-GB retrieval

charge

Archive data

Long term archive

data

S3 Glacier

Instant

Retrieval

Rarely

accessed data

New

S3 Glacier

Flexible Retrieval

(formerly S3 Glacier)

• Milliseconds

access

• Per-GB retrieval

charge

• Retrieval options

from minutes to

hours

• Free bulk

retrievals

• Retrieval in hours

• Milliseconds access

• Per-GB retrieval

charge

• Milliseconds access

Your choice of Amazon S3 storage classes

AWS Region  3 Availability Zones

S3 Standard

S3 Standard-IA

S3 One

Zone-IA

S3 Intelligent-

Tiering

S3 Glacier

Deep Archive

Re-creatable,

infrequently accessed

data

S3 Outposts

On-premises data

Frequently

accessed data

• Milliseconds

access

• No retrieval

charge

Changing access

patterns

• Milliseconds

access

• No retrieval

charge

• Archive Instant

Access tier

Infrequently

accessed data

• Milliseconds

access

• Per-GB retrieval

charge

Archive data

Long term archive

data

S3 Glacier

Instant

Retrieval

Rarely

accessed data

New

S3 Glacier

Flexible Retrieval

(formerly S3 Glacier)

• Milliseconds

access

• Per-GB retrieval

charge

• Retrieval options

from minutes to

hours

• Free bulk

retrievals

• Retrieval in hours

• Milliseconds access

• Per-GB retrieval

charge

• Milliseconds access

New

Your choice of Amazon S3 storage classes

S3 Standard

S3 Standard-IA

S3 One

Zone-IA

S3 Intelligent-

Tiering

S3 Glacier

Deep Archive

Re-creatable,

infrequently

accessed data

S3 Outposts

On-premises data

AWS AZ AWS Outposts

Frequently

accessed data

• Milliseconds

access

• No retrieval

charge

Changing access

patterns

• Milliseconds

access

• No retrieval

charge

• Archive Instant

Access tier

Infrequently

accessed data

• Milliseconds

access

• Per-GB retrieval

charge

Archive data

Long term archive

data

S3 Glacier

Instant

Retrieval

Rarely

accessed data

New

S3 Glacier

Flexible Retrieval

(formerly S3 Glacier)

• Milliseconds

access

• Per-GB retrieval

charge

• Retrieval options

from minutes to

hours

• Free bulk

retrievals

• Retrieval in hours

• Milliseconds access

• Per-GB retrieval

charge

• Milliseconds access

Your choice of Amazon S3 storage classes

S3 Standard

S3 Standard-IA

S3 One

Zone-IA

S3 Intelligent-

Tiering

S3 Glacier

Deep Archive

Re-creatable,

infrequently

accessed data

S3 Outposts

On-premises data

AWS AZ AWS Outposts

Frequently

accessed data

• Milliseconds

access

• No retrieval

charge

Changing access

patterns

• Milliseconds

access

• No retrieval

charge

• Archive Instant

Access tier

Infrequently

accessed data

• Milliseconds

access

• Per-GB retrieval

charge

Archive data

Long term archive

data

S3 Glacier

Instant

Retrieval

Rarely

accessed data

New

S3 Glacier

Flexible Retrieval

(formerly S3 Glacier)

• Milliseconds

access

• Per-GB retrieval

charge

• Retrieval options

from minutes to

hours

• Free bulk

retrievals

• Retrieval in hours

• Milliseconds access

• Per-GB retrieval

charge

• Milliseconds access

New

Economics and primary factors to consider

Frequency of access Duration of storage

Uploading data to S3 Glacier Instant Retrieval

Specify GLACIER_IR in the PUT API request header

PUT /my-image.jpg HTTP/1.1

Host: myBucket.s3.<Region>.amazonaws.com

Date: Wed, 1 Dec 2021 17:50:00 GMT

Authorization: authorization string

Content-Type: image/jpeg

Content-Length: 11434

Expect: 100-continue

x-amz-storage-class: GLACIER_IR

Milliseconds access

Lifecycle data with predictable access patterns

S3 Glacier

Flexible Retrieval

S3 Standard S3 Standard-IA

S3 Glacier

Deep Archive

Minutes to hours

S3 Glacier Instant

Retrieval

New

Fine tune your Lifecycle policies

Lifecycle rules take action based on object age:

1. Move objects older than 90 days to S3 Glacier Instant Retrieval

2. Move objects older than 365 days to S3 Glacier Deep Archive

U S E L I F E C Y C L E P O L I C I E S T O T R A N S I T I O N O B J E C T S T O A N O T H E R S T O R A G E C L A S S

S3 Standard

S3 Glacier

Instant Retrieval

S3 Glacier Deep

Archive

New

With S3 Glacier Instant Retrieval, we are able to lower

storage costs for imaging workloads that require data to

be immediately accessible when requested, such as when

a radiologist needs to review a patient’s prior history or

a collection of archived images needs to be analyzed.

Daniel Ostrow

Chief Enterprise Architect @ Intelerad

The S3 Glacier Instant Retrieval storage class

provides low storage cost for the NASCAR Library

which houses our growing media archives, and

enables our content creators to interact with data of

any age in near real time.

Chris Wolford

Senior Director of Media & Event Technology @ NASCAR

S3 Intelligent-Tiering

$250,000,000

Since the launch of S3 Intelligent-Tiering,

customers storage cost savings now exceed

$250,000,000

Since the launch of S3 Intelligent-Tiering,

customers storage cost savings now exceed

Default storage class for data lakes, data

analytics, and new applications

• Only cloud storage that delivers automatic storage cost savings

• Moves objects between three access tiers for a small monthly

monitoring and automation fee

• New Archive Instant Access tier that delivers up to 68% lower

cost, without any impact, without any impact on performance

• No operational overhead and no retrieval fees

• Designed for 99.9% availability and 99.999999999% (11 9s) of

durability

Milliseconds access (automatic)

Minutes to hours (optional)

Frequent

Access tier

Infrequent

Access tier

Archive

Instant

Access tier

Deep

Archive

Access tier

Use S3 Intelligent-Tiering for data with

unknown or changing access patterns

Archive

Access tier

New

Frequent

Access tier

Infrequent

Access tier

Before: S3 Intelligent-Tiering automatically

optimized cost in two access tiers

Now: S3 Intelligent-Tiering automatically

optimizes cost in three access tiers

Frequent

Access tier

Infrequent

Access tier

Archive

Instant

Access tier

+30 days +60 days

Automatically save up to

68% with new Archive

Instant Access

New

Why we built the Archive Instant Access tier

Petabytes of

data stored for indefinite

periods of time

Subsets of data become

rarely accessed for long

periods of time

Data needs millisecond

retrievals when access

patterns change

S3 Intelligent-Tiering: Archive Instant Access

68% lower cost

than the Infrequent

Access tier

Millisecond access

and high

throughput

performance

Automatic savings for

rarely accessed data

Designed for

99.999999999%

durability

S3 Intelligent-Tiering pricing enhancements

Small objects not monitored

or auto-tiered

Optimizing for short-lived

objects

Use S3 Intelligent-Tiering without

needing to analyze object size

distributions

Use S3 Intelligent-Tiering without

needing to analyze what the expected

average life of objects will be

New

Upload directly to S3 Intelligent-Tiering

Specify INTELLIGENT_TIERING in the PUT API request header

PUT /my-image.jpg HTTP/1.1

Host: myBucket.s3.<Region>.amazonaws.com

Date: Wed, 1 Dec 2021 17:50:00 GMT

Authorization: authorization string

Content-Type: image/jpeg

Content-Length: 11434

Expect: 100-continue

x-amz-storage-class: INTELLIGENT_TIERING

S3 Intelligent-Tiering continuous optimization

S3 Intelligent-

Tiering

(2018)

S3 Intelligent-Tiering

Async Archive Access

tiers (2020)

S3 Intelligent-

Tiering Pricing

optimizations (2021)

S3 Intelligent-

Tiering Archive

Instant Access

(2021)

S3 Intelligent-

Async Archive Events

(2021)

Inventory reports

by Access tiers

(2019)

AWS Storage Gateway

adds S3 Intelligent-

Tiering (2019)

New

Since the launch of S3 Intelligent-Tiering in 2018,

we’ve automatically saved ~30% per month on our

storage costs without any impact on performance

or need to analyze our data. With the new Archive

Instant Access tier, we anticipate automatically

realizing the benefit of archive storage pricing,

while retaining the ability to access our data

instantly when needed.

Kalyana Chadalavada

Head of Efficiency @ Stripe

Using S3 Intelligent-Tiering, we can implement storage changes

without interruptions to service and activity. Our data is automatically

moved to lower-cost tiers based on data access, saving us a lot of

development time in addition to reducing costs. With that time, my

team can focus on identifying other opportunities to reduce

infrastructure costs in support of our organizational goals. The new

access tier in S3 Intelligent-Tiering will help us save even more on

storage costs.

Joshua Bergen

Cost Management Lead @ Epic Games

We wanted to find a way to quickly optimize storage costs across the

largest and fastest growing buckets across the enterprise. Because the

storage usage patterns vary widely across our top buckets, there was

no clear-cut rule we could safely apply without taking on some

operational overhead. The S3 Intelligent-Tiering storage class

delivered automatic storage savings based on the changing access

patterns of our data without impact on performance. We look forward

to S3 Intelligent-Tiering’s new access tier which will allow us to realize

even greater savings without additional effort.

Jerzy Grzywinski

Director of Software Engineering @ Capital One

Define your archive strategy

Content is growing rapidly, your storage cost

don’t have to

Media User data Files Connected

devices

Automatic archiving based on

the last access of your objects

Fastest access to

archive storage

S3 Glacier is the best place to archive your data

Lowest storage cost

in the cloud

Flexible retrieval options

S3 Glacier is the best place to archive

Storage cost

$0.004 per GB-month $0.0036 per GB-month $0.00099 per GB-month

Data retrieval

Milliseconds retrieval

Expedited: 1-5 minutes

Standard: 3-5 hours

FREE Bulk: 5-12 hours

Standard: Within 12 hours

Bulk: Within 48 hours

Minimum object

duration

90 days 90 days 180 days

S3 Glacier

Instant Retrieval

New

S3 Glacier

Flexible Retrieval

S3 Glacier

Deep Archive

S3 Lifecycle filters and actions

• Even more granular S3 Lifecycle configurations

• Object size filters to optimize transition costs

and save more on storage

• Number of noncurrent versions to optimize

storage spend for versioned buckets

New

What if you have unknown

or changing access patterns?

Get the lowest storage cost for data that is

not being accessed

Frequent

Access tier

Infrequent

Access tier

+30

days

+90

days

Deep

Archive

Access tier

Archive

Access tier

+185

days

Milliseconds access (automatic)

Minutes to hours (optional)

Up to 72% storage cost

savings using the Archive

Access

Archive

Instant

Access tier

New

+60

days

Get the lowest storage costs in the cloud

automatically

Frequent

Access tier

Infrequent

Access tier

Archive

Instant

Access tier

+30 days +60 days

Deep

Archive

Access tier

Milliseconds access (automatic) Hours (optional)

Up to 95% storage cost

savings using Deep Archive

Access

+275 days

New

Archive data when you are ready

Configure the Archive Access or

Deep Archive Access tiers, or both

90-day minimum days of

consecutive no access to Archive

Access tier

180-day minimum days of

consecutive no access to Deep

Archive Access tier

Extend the last access time for

archiving up to two years

Amazon S3 Intelligent-Tiering

E N A B L E E V E N T N O T I F I C A T I O N S T O T R A C K O B J E C T S M O V E D T O A S Y N C A R C H I VE

Frequent

Access tier

Infrequent

Access tier

Archive

Instant

Access tier

Deep

Archive

Access tier

New

Fundamentally changes archive storage

MILLISECONDS

RETRIEVAL

S3 Glacier Instant

Retrieval

ACCESSING IN

DATA LAKE

Archive Instant Access

tier in S3 Intelligent-

Tiering

Data is ready

when you need it

Data is

put to work

Dynamically

cost optimized

THE LOWEST

COST

S3 Glacier Deep

Archive

FREE BULK

RETRIEVALS

S3 Glacier Flexible

Retrievals

Data

Backup

1

2

3

How Splunk uses Amazon S3

to optimize cost and turn

data into insights

Srinivas Bobba

Principal Product Manager, Splunk Inc

Splunk turns data into business outcomes

Unlocking business value by turning data into doing

70

%

Lower risk of data

breach, IP theft,

and fraud

90

%

Observability

82

%

IT Operations

Platform for all

your data needs

1

Security

Accelerate development

times by 90% when you

bring data to DevOps

Reduce incidents and

downtime with Splunk’s

AIOps platform

Splunk uses Amazon S3 to turn data into

insights

Easily index machine data and drive insights across multiple teams in your

organization

Customers save up

to ~75%* in

infrastructure cost

after enabling

SmartStore

More than 200 PB

of machine data

stored in Amazon

S3 across all

Regions

10s of millions of

searches on a daily

basis

Wide range of use

cases with variables

access patterns

*Reference only, savings may

vary based on configurations

Splunk architecture with SmartStore

Decoupled Compute and Storage

Hot/Warm

Storage

Cold

Storage

Classic Architecture

S3 Storage

SmartStore Architecture

Hot/Cache

Storage

Hot/Cache

Storage

Indexer Search

Clustered Indexer

Local Storage (cache)

Cache Management Layer

manage cache space

event data

copy warm buckets to

object storage

move buckets from

object storage to local

storage

write & read buckets

locally

replicate hot buckets

for fault tolerance

Splunk SmartStore data flow

Search Head

Amazon S3

Data Lifecycles with Splunk Cloud Platform

Index

Active Storage (DDAS) –

Searchable

(S3 Standard)

DDAA

Archive Storage

(S3 Glacier Flexible

Retrieval)

- Splunk managed

Ingest

Archiving & Restore

DDSS

Self-Storage (S3 Glacier

Flexible Retrieval)

Customer managed

Splunk Cloud Platform

Aged Data

Key takeaways

Decouple storage and compute to sustain data growth

Use S3 Intelligent-Tiering for data with changing access

patterns

For compliance and data retention, use S3 Glacier

storage classes to manage data lifecycle

Benefit from lower cost and ability to focus on time to

market, launching new features to customers

1

2

3

4

Thank you!

Jessie Felix

[email protected]om

Andrew Kutsy

[email protected]

Srinivas Bobba