Best Practices to Configure S3 buckets

Since its release in 2006, AWS S3 service has seen hunderds of changes, and yesterday’s best practices might no longer be perfect today. We’ll go over a number of best practices to make it simpler to secure and operate your data stores.

New-style permissions

Traditionally, permittions for S3 are controlled by IAM role, bucket policy, object ownership, bucket ACL and object ACL. In particular, the interaction between policies and ACLs are not intuitive, and often paradoxical. For the worst example, you can have a bucket from account A contain objects owned by account B and no way to access them.

Fortunately, it is now possible to disable the ACLs entirely by setting the bucket’s S3 Object Ownership setting to BucketOwnerEnforced. This setting completely disables ACLs, and bucket policy becomes the only mechanism for permissions.

The new option is on by default for new buckets, but for existing buckets, you might want to explicitly change the settings. You can read more details in the documentation

Versioning

Object versioning is a mechanism to keep old versions of objects as they are overwritten or deleted. If you overwrite an object, the previous content is kept as “non-current object version”. If you delete an object, instead of removing any trace of it, S3 creates a “delete marker” and the previous content is kept, again as “non-current object version”.

For any buckets with business critical data, this feature is a life-saver. It allows to recover from accidental deletion of data, as well of bugs that compute wrong data. Therefore, it’s wise to enable it by default.

Lifecycle rules

Lifecycle rules allow to automatically delete S3 objects, or move them to cheaper storage tiers.

The first important use of lifecycle rules is connected to versioning. We generally don’t want to keep all versions of an object forever. Instead, after some time, sufficient to detect any problems, old versions can be deleted. Lifecycle rules can accomplish that automatically. If your bucket has versioning enabled, it always should have a rule to delete old objects. Typically, after old versions are expired, you want to expire the delete markers, too.

The second typical usage is to move objects to storage tiers. For example, any raw data that is not normally used, but must be kept forever can be moved to the Glacier storage level. It is a good idea to consider in many cases. For details, please see official storage tiers documentation.

Encryption

There are three reasons why you might want to enable encryption for S3 buckets

• To accomodate security standards and auditors
• To account for catastrophic natural events, where S3 disk will be lying on the ground unprotected
• To safeguard against security compromise at AWS

By default, all new S3 buckets are encrypted using S3-managed key (called SSE-S3). What this means is that the S3 service generates an AES encryption key, and uses it to encrypt the data as its written to the physical disks. This encryption can be reasonable enabled for all buckets, since it has very low overhead, and checks two of the three possible reasons above.

It might be reasonable to use S3 server-side encryption, but store a key in AWS KMS service. This approach is called SSE-KMS. It gives some additional security, but KMS is an expensive service, so using it for many small objects can cost hundreds of dollars per day. Additional option called bucket key can dramatically reuse the cost by sharing the key between objects created around the same time.

The best practice is to use either of the two options - SSE-S3, or SSE-KMS with bucket key.

For completeness, we’ll note that S3 also support client-side encryption, but it is only necessary in case you don’t trust AWS, or not permitted to trust AWS, and these are rare cases.

Server-side access logging

S3 can be configured to log every request to a bucket. These logs can be very valuable in diagnosing performance troubles, finding unused data, and recovering from an accident, or auditing accesses.

For example, if a particular analytics workload is running slow, access logs allow to see exactly how long each operation takes, specifically time to first byte and download time. Often, just looking at this numbers is sufficient to understand the problem.

For another example, if a data was corrupted, access logs allow to understand what code did that at what time, and therefore pin-point the problem.

We’d recommend that server-side access logging be always enabled.

Bucket inventory

Another powerful tool is bucket inventory. With it in place, S3 can create a list of every object in a bucket and store it in Parquet format that you can process with Spark, or Pandas.

The obvious use of bucket inventory is various cost optimization - it’s easy to explore the different paths and see which coonsume unexpected amount of storage. It can be even combined with access logs to find data that is large and unused.

It is also super helpful in recoverying from accidents - say, if you delete something by mistake, you can consult inventory to find a list of objects you deleted, and, in a versioned bucket, issue API requests to restore those objects.

Just like with access logging, the inventory size and cost is miniscule compared to the size of the actual data, and we’d recommend enabling it for all buckets.

Complete example with Terraform

Let’s work together a complete example of creating a new S3 bucket using the recommendations above. We’ll assume we have a bucket acme-data that will contain the actual data, and that we already have another bucket, acme-logs, that will contain the logs. The data bucket can be created as follows

resource "aws_s3_bucket" "acme-data" {
  bucket = "acme-data"

  tags = {
    Name = "acme-data"
  }
} 

// Disable old confusing ACL mechanim for this bucket. 
resource "aws_s3_bucket_ownership_controls" "acme-data" {
  bucket = aws_s3_bucket.acme-data.id
  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

// Enable versioning.
resource "aws_s3_bucket_versioning" "acme-data" {
  bucket = aws_s3_bucket.acme-data.id
  versioning_configuration {
    status = "Enabled"
  }
}

// Add lifecycle rule to remove old versions after a while
resource "aws_s3_bucket_lifecycle_configuration" "acme-data" {
  bucket = aws_s3_bucket.acme-data.id

  rule {
    id = "remove old versions"
    status = "Enabled"

    // Usually, two weeks are enough to detect any data corruption or
    // unplanned deletions. After that, expire old versions
    noncurrent_version_expiration {
      noncurrent_days = 14
    }

    // Expire delete marker after non-current versions have expired
    expiration {
        expired_object_delete_marker = true
    }

    // Remove left-over objects from multi-part uploads that were not completed
    abort_incomplete_multipart_upload {
        days_after_initiation = 1
    }
  }
}

resource "aws_s3_bucket_logging" "acme-data" {
  bucket = aws_s3_bucket.acme-data.id

  // Use string bucket name to simplify the example.
  target_bucket = "acme-logs"
  target_prefix = "s3/acme-data"
}

resource "aws_s3_bucket_inventory" "acme-data" {
  bucket = aws_s3_bucket.acme-data.id
  name   = "ParquetDaily"

  included_object_versions = "All"
  optional_fields = ["Size", "LastModifiedDate", "StorageClass", "IntelligentTieringAccessTier"]

  schedule {
    frequency = "Daily"
  }

  destination {
    bucket {
      format     = "Parquet"
      bucket_arn = "arn:aws:s3:::acme-logs"
      prefix     = "s3-inventory"
    }
  }
}

For completeness, let’s show how the logging bucket itself can be configured. Here, we will not setup versioning, inventory and logging, because the bucket will only be written by AWS services.

resource "aws_s3_bucket" "acme-logs" {
  bucket = "acme-logs"

  tags = {
    Name = "acme-logs"
  }
}

resource "aws_s3_bucket_ownership_controls" "acme-logs" {
  bucket = aws_s3_bucket.acme-logs.id
  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

resource "aws_s3_bucket_policy" "acme-logs-policy" {
  bucket = aws_s3_bucket.acme-logs.id
  policy = data.aws_iam_policy_document.acme-logs-policy.json
}

data "aws_iam_policy_document" "acme-logs-policy" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["logging.s3.amazonaws.com"]
    }

    resources = ["arn:aws:s3:::acme-logs/*"]

    actions = [
      "s3:PutObject"
    ]

    effect = "Allow"
  }
}

The above definitions can be easily copy-pasted, with acme-data and acme-logs replaced to your liking.

Conclusion

In this post, we disucssed a few recommendation for configuring your S3 buckets. We hope it will be useful in your next project.