Skip to content

AWS DataSync

Service Overview and Purpose

AWS DataSync is a data transfer service that simplifies, automates, and accelerates moving large amounts of data between on-premises storage systems and AWS storage services, as well as between AWS storage services. DataSync automatically handles many of the tasks related to data transfers that can slow down migrations or burden IT operations.

Key Characteristics

  • One-Time or Scheduled: Support for both migration and ongoing sync
  • High Performance: Up to 10x faster than open-source tools
  • Data Validation: Automatic verification and integrity checks
  • Network Optimization: Built-in network optimization and bandwidth throttling
  • Security: End-to-end encryption and AWS IAM integration

Key Features and Capabilities

Supported Data Sources and Destinations

Sources

  1. On-Premises Storage
  2. NFS: Network File System (v3, v4.0, v4.1)
  3. SMB: Server Message Block (v2.1, v3.0, v3.02, v3.1.1)
  4. HDFS: Hadoop Distributed File System

  5. Object Storage: Self-managed object storage

  6. AWS Storage Services

  7. S3: Any storage class
  8. EFS: Elastic File System
  9. FSx: Windows File Server, Lustre, NetApp ONTAP, OpenZFS

  10. Snowcone: Edge computing device

  11. Other Cloud Providers

  12. Google Cloud Storage
  13. Azure Blob Storage
  14. Azure Files

Destinations

  • All AWS storage services listed above
  • Cross-region transfers within AWS
  • Cross-account transfers

Core Features

Data Transfer Capabilities

  • Incremental Transfer: Only changed data transferred after initial sync
  • Bandwidth Throttling: Control network utilization
  • Parallel Transfers: Multiple parallel connections
  • Compression: Automatic data compression during transfer
  • Sparse File Support: Efficient handling of sparse files

Data Validation and Integrity

  • Checksum Verification: End-to-end data integrity
  • Transfer Logs: Detailed logging of all operations
  • Error Handling: Automatic retry mechanisms
  • Verification Reports: Post-transfer validation reports

Scheduling and Automation

  • Flexible Scheduling: Hourly, daily, weekly schedules
  • CloudWatch Events: Event-driven automation
  • Task Management: Create, monitor, and manage transfer tasks
  • API Integration: Programmatic control and automation

Security Features

  • Encryption in Transit: TLS 1.2 encryption
  • VPC Endpoints: Private network access
  • IAM Integration: AWS identity and access management
  • Data Filtering: Include/exclude patterns

Use Cases and Scenarios

Primary Use Cases

  1. Data Migration to AWS
  2. Lift and Shift: Migrate existing file systems to AWS
  3. Application Modernization: Move data as part of app migration
  4. Data Center Exit: Complete data center migration

  5. Disaster Recovery: Establish cloud-based DR

  6. Hybrid Cloud Architectures

  7. Data Distribution: Distribute data across locations
  8. Content Distribution: Replicate content to edge locations
  9. Backup and Archive: Regular backup to cloud storage

  10. Multi-Site Synchronization: Keep multiple sites in sync

  11. Data Processing Workflows

  12. Analytics Pipelines: Move data for analytics processing
  13. Machine Learning: Transfer datasets for ML workflows
  14. Media Workflows: Move media files for processing

  15. Compliance: Replicate data for compliance requirements

  16. Edge to Cloud Transfer

  17. IoT Data: Transfer IoT data from edge to cloud
  18. Remote Sites: Sync data from remote locations
  19. Branch Offices: Centralize branch office data
  20. Field Data Collection: Transfer field-collected data

Architecture Patterns

  1. One-Time Migration
  2. Initial data assessment
  3. Bandwidth planning
  4. Migration execution

  5. Validation and cutover

  6. Ongoing Synchronization

  7. Scheduled sync tasks
  8. Event-driven synchronization
  9. Incremental updates

  10. Conflict resolution

  11. Multi-Tier Data Movement

  12. Hot data to S3 Standard
  13. Warm data to S3 IA
  14. Cold data to Glacier
  15. Automated lifecycle management

Pricing Models and Cost Optimization

Pricing Components

  1. Data Transfer Charges
  2. Per GB transferred: $0.0125 per GB (first 50TB)
  3. Volume discounts: Lower rates for higher volumes
  4. No charges: For data already in AWS (between services)

  5. Cross-region: Additional charges for cross-region transfers

  6. Agent Costs

  7. No software licensing: DataSync agent is free
  8. Infrastructure costs: EC2 instance costs if deployed on EC2

  9. On-premises: No additional costs for on-premises deployment

  10. Storage Costs

  11. Destination storage: Standard AWS storage pricing
  12. Temporary storage: No charges for temporary staging
  13. Bandwidth: Network bandwidth costs if applicable

Cost Optimization Strategies

  1. Transfer Optimization 

    # Monitor transfer efficiency
aws datasync describe-task-execution \
  --task-execution-arn arn:aws:datasync:region:account:task/task-id/execution/exec-id \
  --query 'BytesTransferred'

  2. Scheduling Optimization

  3. Transfer during off-peak hours
  4. Use bandwidth throttling
  5. Optimize transfer frequency

  6. Batch smaller transfers

  7. Storage Class Optimization

  8. Transfer directly to appropriate S3 storage class
  9. Use Intelligent-Tiering for unknown access patterns
  10. Implement lifecycle policies post-transfer

  11. Consider regional storage costs

  12. Network Optimization

  13. Use VPC endpoints to avoid internet charges
  14. Optimize agent placement
  15. Use compression when beneficial
  16. Monitor and adjust bandwidth settings

Configuration Details and Best Practices

DataSync Agent Deployment

# Deploy agent on EC2 (using CloudFormation)
aws cloudformation create-stack \
  --stack-name datasync-agent \
  --template-body file://datasync-agent-template.yaml \
  --parameters ParameterKey=InstanceType,ParameterValue=m5.2xlarge \
               ParameterKey=SubnetId,ParameterValue=subnet-12345678 \
               ParameterKey=SecurityGroupId,ParameterValue=sg-12345678

# Activate agent
aws datasync create-agent \
  --agent-name "Production-DataSync-Agent" \
  --activation-key "activation-key-from-console" \
  --security-group-arns arn:aws:ec2:region:account:security-group/sg-12345678 \
  --subnet-arns arn:aws:ec2:region:account:subnet/subnet-12345678 \
  --tags Key=Environment,Value=Production

Location Configuration

# Create NFS location
aws datasync create-location-nfs \
  --server-hostname 192.168.1.100 \
  --subdirectory /data/exports/files \
  --on-prem-config '{
    "AgentArns": [
      "arn:aws:datasync:region:account:agent/agent-12345678"
    ]
  }' \
  --mount-options '{
    "Version": "NFS4_1"
  }' \
  --tags Key=LocationType,Value=Source

# Create S3 location
aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::my-migration-bucket \
  --s3-config '{
    "BucketAccessRoleArn": "arn:aws:iam::account:role/DataSyncS3Role"
  }' \
  --s3-storage-class STANDARD_IA \
  --tags Key=LocationType,Value=Destination

Task Configuration

# Create DataSync task
aws datasync create-task \
  --source-location-arn arn:aws:datasync:region:account:location/loc-source123 \
  --destination-location-arn arn:aws:datasync:region:account:location/loc-dest456 \
  --name "Production-File-Migration" \
  --options '{
    "VerifyMode": "POINT_IN_TIME_CONSISTENT",
    "OverwriteMode": "ALWAYS",
    "Atime": "BEST_EFFORT",
    "Mtime": "PRESERVE",
    "Uid": "INT_VALUE",
    "Gid": "INT_VALUE",
    "PreserveDeletedFiles": "PRESERVE",
    "PreserveDevices": "NONE",
    "PosixPermissions": "PRESERVE",
    "BytesPerSecond": 104857600,
    "TaskQueueing": "ENABLED",
    "LogLevel": "TRANSFER"
  }' \
  --excludes '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "*.tmp"
  }' \
  --includes '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "/important/*"
  }' \
  --tags Key=MigrationType,Value=Production Key=Priority,Value=High

Best Practices

  1. Agent Sizing and Placement
  2. Compute Resources: Minimum 4 vCPUs, 8GB RAM
  3. Network: 10 Gbps network for high-throughput transfers
  4. Storage: Local storage for temporary staging

  5. Placement: Close to source data for optimal performance

  6. Network Configuration

  7. Dedicated Network: Use dedicated network paths when possible
  8. Bandwidth Management: Configure appropriate bandwidth limits
  9. Security Groups: Proper firewall configuration

  10. VPC Endpoints: Use for private connectivity

  11. Transfer Optimization

  12. File System Tuning: Optimize source file system performance
  13. Concurrent Transfers: Use multiple tasks for large datasets
  14. Scheduling: Transfer during off-peak hours

  15. Monitoring: Continuous monitoring of transfer performance

  16. Security Configuration

  17. IAM Roles: Least privilege access principles
  18. Encryption: Enable encryption in transit
  19. Network Security: Use private networking when possible
  20. Access Logging: Enable CloudTrail for audit

Task Scheduling

# Create scheduled task using CloudWatch Events
aws events put-rule \
  --name "DataSync-Daily-Backup" \
  --schedule-expression "cron(0 2 * * ? *)" \
  --description "Daily DataSync backup at 2 AM"

aws events put-targets \
  --rule "DataSync-Daily-Backup" \
  --targets Id=1,Arn=arn:aws:datasync:region:account:task/task-12345678,RoleArn=arn:aws:iam::account:role/DataSyncExecutionRole

Integration with Other AWS Services

Storage Service Integration

  1. S3 Integration 

    # Create location with S3 storage class
aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::analytics-data \
  --s3-storage-class GLACIER \
  --s3-config '{
    "BucketAccessRoleArn": "arn:aws:iam::account:role/DataSyncS3Role"
  }'

  2. EFS Integration 

    # Create EFS location
aws datasync create-location-efs \
  --efs-filesystem-arn arn:aws:elasticfilesystem:region:account:file-system/fs-12345678 \
  --ec2-config '{
    "SubnetArn": "arn:aws:ec2:region:account:subnet/subnet-12345678",
    "SecurityGroupArns": [
      "arn:aws:ec2:region:account:security-group/sg-12345678"
    ]
  }' \
  --access-point-arn arn:aws:elasticfilesystem:region:account:access-point/fsap-12345678

  3. FSx Integration 

    # Create FSx Windows location
aws datasync create-location-fsx-windows \
  --fsx-filesystem-arn arn:aws:fsx:region:account:file-system/fs-12345678 \
  --security-group-arns arn:aws:ec2:region:account:security-group/sg-12345678 \
  --user Administrator \
  --domain corp.example.com \
  --password SecurePassword123!

Automation and Orchestration

# Lambda function for automated DataSync execution
import boto3
import json

def lambda_handler(event, context):
    datasync = boto3.client('datasync')

    # Start task execution
    response = datasync.start_task_execution(
        TaskArn='arn:aws:datasync:region:account:task/task-12345678',
        OverrideOptions={
            'VerifyMode': 'POINT_IN_TIME_CONSISTENT',
            'LogLevel': 'TRANSFER'
        }
    )

    return {
        'statusCode': 200,
        'body': json.dumps({
            'TaskExecutionArn': response['TaskExecutionArn']
        })
    }

Monitoring Integration

# Create CloudWatch alarm for task failures
aws cloudwatch put-metric-alarm \
  --alarm-name "DataSync-Task-Failures" \
  --alarm-description "Alert on DataSync task failures" \
  --metric-name "TaskExecutionsFailed" \
  --namespace "AWS/DataSync" \
  --statistic Sum \
  --period 3600 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --dimensions Name=TaskId,Value=task-12345678 \
  --alarm-actions arn:aws:sns:region:account:datasync-alerts

Workflow Integration

  • Step Functions: Orchestrate complex data workflows
  • AWS Batch: Coordinate with batch processing jobs
  • SageMaker: Automate ML dataset preparation
  • EMR: Trigger big data processing after transfer

Security Considerations

Data Protection

  1. Encryption in Transit
  2. TLS 1.2: All data encrypted during transfer
  3. Certificate Management: Automatic certificate handling
  4. VPC Endpoints: Private network encryption

  5. Protocol Security: Secure protocol implementations

  6. Access Control 

    {
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "datasync:DescribeTask",
      "datasync:StartTaskExecution"
    ],
    "Resource": "arn:aws:datasync:*:*:task/task-*",
    "Condition": {
      "StringEquals": {
        "aws:RequestedRegion": ["us-west-2", "us-east-1"]
      }
    }
  }]
}

  7. Network Security

  8. VPC Deployment: Deploy agents in private subnets
  9. Security Groups: Restrict network access
  10. NACLs: Additional network layer security
  11. VPC Endpoints: Private API access

Compliance and Auditing

  1. Audit Trails 

    # Enable CloudTrail for DataSync operations
aws cloudtrail create-trail \
  --name "DataSync-Audit-Trail" \
  --s3-bucket-name "audit-logs-bucket" \
  --include-global-service-events \
  --is-multi-region-trail \
  --enable-log-file-validation

  2. Data Validation

  3. Checksum Verification: Automatic integrity checks
  4. Transfer Logs: Detailed operation logging
  5. Error Reporting: Comprehensive error tracking

  6. Validation Reports: Post-transfer verification

  7. Compliance Features

  8. HIPAA Compliance: Healthcare data transfers
  9. PCI DSS: Payment card data protection
  10. GDPR: European data protection compliance
  11. SOC Compliance: Service organization controls

Monitoring and Troubleshooting

CloudWatch Metrics

Task Metrics

  • TaskExecutionsStarted: Number of task executions started
  • TaskExecutionsSucceeded: Successfully completed executions
  • TaskExecutionsFailed: Failed task executions
  • FilesTransferred: Number of files transferred
  • BytesTransferred: Amount of data transferred

Performance Metrics

  • ThroughputPerSecond: Data transfer rate
  • FilesPerSecond: File transfer rate
  • ErrorRate: Percentage of transfer errors
  • Duration: Task execution duration

Agent Metrics

  • AgentOnline: Agent availability status
  • NetworkUtilization: Network bandwidth usage
  • CPUUtilization: Agent CPU usage
  • MemoryUtilization: Agent memory usage

Monitoring Setup

# Create comprehensive monitoring dashboard
aws cloudwatch put-dashboard \
  --dashboard-name "DataSync-Monitoring" \
  --dashboard-body '{
    "widgets": [{
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/DataSync", "BytesTransferred", "TaskId", "task-12345678"],
          [".", "FilesTransferred", ".", "."],
          [".", "ThroughputPerSecond", ".", "."]
        ],
        "period": 300,
        "stat": "Sum",
        "region": "us-west-2",
        "title": "DataSync Transfer Performance"
      }
    }]
  }'

# Set up task completion notification
aws sns create-topic --name "DataSync-Notifications"
aws events put-rule \
  --name "DataSync-Task-Completion" \
  --event-pattern '{
    "source": ["aws.datasync"],
    "detail-type": ["DataSync Task Execution State Change"],
    "detail": {
      "State": ["SUCCESS", "ERROR"]
    }
  }'

Common Issues and Solutions

  1. Performance Issues
  2. Network Bandwidth: Verify available bandwidth
  3. Agent Resources: Check CPU and memory utilization
  4. File System Performance: Optimize source file system

  5. Concurrent Operations: Limit concurrent transfers

  6. Connectivity Issues

  7. Agent Communication: Verify agent registration
  8. Network Configuration: Check security groups and NACLs
  9. DNS Resolution: Verify hostname resolution

  10. Port Accessibility: Ensure required ports are open

  11. Transfer Failures

  12. Permissions: Verify IAM roles and policies
  13. Storage Capacity: Check destination storage capacity
  14. File Locks: Release locked files on source
  15. Network Interruptions: Monitor network stability

Troubleshooting Commands

# Check task execution details
aws datasync describe-task-execution \
  --task-execution-arn arn:aws:datasync:region:account:task/task-id/execution/exec-id

# List task executions with filtering
aws datasync list-task-executions \
  --task-arn arn:aws:datasync:region:account:task/task-12345678 \
  --max-results 10

# Verify agent status
aws datasync describe-agent \
  --agent-arn arn:aws:datasync:region:account:agent/agent-12345678

# Check location connectivity
aws datasync describe-location-nfs \
  --location-arn arn:aws:datasync:region:account:location/loc-12345678

# Monitor transfer progress
aws datasync describe-task \
  --task-arn arn:aws:datasync:region:account:task/task-12345678 \
  --query 'Status'

Exam-Specific Tips and Common Scenarios

Key Exam Topics

  1. Use Case Selection
  2. Migration vs synchronization scenarios
  3. One-time vs ongoing transfer requirements
  4. Performance vs cost considerations

  5. Supported protocols and systems

  6. Architecture Design

  7. Agent placement and sizing
  8. Network configuration and optimization
  9. Security and compliance requirements

  10. Integration with other AWS services

  11. Performance Optimization

  12. Bandwidth management and throttling
  13. Concurrent transfer strategies
  14. Network optimization techniques
  15. Transfer scheduling and automation

Common Exam Scenarios

  1. Data Center Migration
  2. Large-scale file system migration
  3. Minimal downtime requirements
  4. Network bandwidth limitations

  5. Validation and cutover strategies

  6. Hybrid Cloud Synchronization

  7. Ongoing data synchronization
  8. Multi-site data distribution
  9. Backup and disaster recovery

  10. Content distribution networks

  11. Analytics Data Pipeline

  12. Regular data ingestion for analytics
  13. Data lake population
  14. ML dataset preparation

  15. Real-time vs batch processing

  16. Compliance and Archival

  17. Long-term data retention
  18. Regulatory compliance requirements
  19. Data integrity and validation
  20. Audit trail maintenance

Exam Tips

  • Know supported protocols and their limitations
  • Understand agent requirements and deployment options
  • Remember performance factors affecting transfer speed
  • Know integration patterns with AWS storage services
  • Understand security features and compliance capabilities

Hands-on Examples and CLI Commands

Complete Migration Workflow

# 1. Create and activate agent
aws datasync create-agent \
  --agent-name "Migration-Agent" \
  --activation-key "ABCDEF123456" \
  --tags Key=Project,Value=DataCenterMigration

# 2. Create source location (NFS)
aws datasync create-location-nfs \
  --server-hostname file-server.company.local \
  --subdirectory /data/shared \
  --on-prem-config '{
    "AgentArns": ["arn:aws:datasync:region:account:agent/agent-12345678"]
  }' \
  --tags Key=LocationType,Value=Source

# 3. Create destination location (S3)
aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::company-data-migration \
  --s3-config '{
    "BucketAccessRoleArn": "arn:aws:iam::account:role/DataSyncS3Role"
  }' \
  --s3-storage-class STANDARD_IA \
  --tags Key=LocationType,Value=Destination

# 4. Create migration task
aws datasync create-task \
  --source-location-arn arn:aws:datasync:region:account:location/loc-source123 \
  --destination-location-arn arn:aws:datasync:region:account:location/loc-dest456 \
  --name "File-Server-Migration" \
  --options '{
    "VerifyMode": "POINT_IN_TIME_CONSISTENT",
    "OverwriteMode": "ALWAYS",
    "PreserveDeletedFiles": "PRESERVE",
    "PosixPermissions": "PRESERVE",
    "BytesPerSecond": 209715200,
    "LogLevel": "TRANSFER"
  }' \
  --excludes '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "*.log"
  }' \
  --tags Key=MigrationType,Value=Production

# 5. Start task execution
aws datasync start-task-execution \
  --task-arn arn:aws:datasync:region:account:task/task-12345678 \
  --override-options '{
    "VerifyMode": "POINT_IN_TIME_CONSISTENT"
  }'

# 6. Monitor execution progress
aws datasync describe-task-execution \
  --task-execution-arn arn:aws:datasync:region:account:task/task-id/execution/exec-id

Advanced Configuration Examples

# Create task with comprehensive filtering
aws datasync create-task \
  --source-location-arn arn:aws:datasync:region:account:location/loc-source123 \
  --destination-location-arn arn:aws:datasync:region:account:location/loc-dest456 \
  --name "Selective-Sync-Task" \
  --includes '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "/production/*"
  }' '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "*.pdf"
  }' \
  --excludes '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "*.tmp"
  }' '{
    "FilterType": "SIMPLE_PATTERN",
    "Value": "/temp/*"
  }' \
  --options '{
    "TaskQueueing": "ENABLED",
    "BytesPerSecond": 104857600,
    "LogLevel": "TRANSFER"
  }'

# Set up automated scheduling with Lambda
aws lambda create-function \
  --function-name "DataSync-Scheduler" \
  --runtime "python3.9" \
  --role "arn:aws:iam::account:role/DataSyncLambdaRole" \
  --handler "lambda_function.lambda_handler" \
  --zip-file "fileb://datasync-scheduler.zip" \
  --environment Variables='{TASK_ARN=arn:aws:datasync:region:account:task/task-12345678}'

This comprehensive DataSync documentation provides detailed coverage of all aspects needed for AWS certification exams, including practical examples and real-world migration scenarios.