Overleaf docs
Plans and pricingTemplatesUser docsGo to Overleaf
On-premises
On-premises
  • Welcome
    • Server Pro vs. Community Edition
  • Release notes
    • Release notes 5.x.x
      • Doc version recovery
    • Release notes 4.x.x
    • Release notes 3.x.x
      • Full project history migration
    • Release notes 2.x.x
    • Release notes 1.x.x
    • Release notes 0.x.x
  • Getting started
    • Before you start
    • Requirements
      • Skills needed
      • Hardware requirements
      • Software requirements
    • Microservices
    • Server Pro infrastructure
    • What is the Overleaf Toolkit?
  • Installation
    • Introduction
    • Using the Toolkit
      • 1: Download the Toolkit
      • 2: Familiarize yourself with the Toolkit
      • 3: Initialize the configuration
      • 4. Choose Community Edition or Server Pro
      • 5. Personalizing your instance
      • 6. Post-installation tasks
    • Air-gapped/offline deployments
    • Upgrading TeX Live
  • Configuration
    • Overleaf Toolkit
      • Files and locations
      • Toolkit settings
      • Environment variables
      • Server Pro-only configuration
        • LDAP
        • SAML 2.0
        • Sandboxed Compiles
        • Git integration
        • Templates
        • Adding LaTeX user help
      • Logging
      • TLS proxy
      • Branding
      • Localization
      • Email delivery
      • Redis
      • S3
  • Maintenance
    • docker-compose.yml to Toolkit migration
    • Upgrading your deployment
    • Data and backups
      • Exporting projects
    • Extending TeX Live
    • Horizontal scaling
    • S3 migration
    • Updating MongoDB
  • User and project management
    • User management
      • Username migration
    • Understanding license usage
    • Project management
  • Support
    • Project limits
    • Troubleshooting
    • Getting help
    • Support guides
      • Using templates as an individual
    • Overleaf user docs
Powered by GitBook
LogoLogo

Discover Overleaf

  • Home
  • Features

Solutions

  • Plans and pricing
  • For universities
  • For business
  • For government

Resources

  • Templates
  • User docs and LaTeX learning
  • Blog

© Overleaf

On this page
  • Requirements
  • Estimate the disk size needed for the migration
  • Migration steps
  • Step 0 shutdown the instance
  • Step 1 rewrite directory layout
  • Step 2 upload the files
  • Step 3 start the instance pointing at S3
  • Rolling back

Was this helpful?

Export as PDF
  1. Maintenance

S3 migration

PreviousHorizontal scalingNextUpdating MongoDB

Last updated 4 months ago

Was this helpful?

These instructions are for v5.x and later. If you are following this guide for an earlier version please use sharelatex instead of overleaf in path names and SHARELATEX_ prefix instead of OVERLEAF_ for environment variables.

We'd love to hear from you! If you'd like to share with us how many files you migrated over, their overall volume, and how long the migration took email .

This guide will walk you through the migration from on-disk storage to an S3-compatible object storage. It refers to sections of the introduction document on the .

Requirements

  • A S3 compatible object storage to talk to, see for options

  • Free disk space for migrating existing data, about the current on disk size

  • A maintenance window for doing the actual migration

  • A full backup, including the config, to enable restoring from it

Estimate the disk size needed for the migration

We can use du for calculating the current disk usage:

docker exec sharelatex \
  du --human-readable --max-depth=0 /var/lib/overleaf/data/user_files

docker exec sharelatex \
  du --human-readable --max-depth=0 /var/lib/overleaf/data/template_files

In case you do not have sufficient disk space available on the current server, try attaching another disk to the server.

The history directories already have the correct layout. You can upload directly from the bind-mounted source folder, which does not require any additional disk space.

Migration steps

Step 0 shutdown the instance

We need to make sure that all user/template files will get migrated. It is best to shut down the instance to avoid missing newly uploaded files.

Please see our guide on performing a consistent a backup for the shutdown procedure.

Step 1 rewrite directory layout

We need to rewrite the directory layout of project files for uploading them to S3. The directory layout for local storage in filestore is <project-id>_<file-id> and the directory layout in S3 is <project-id>/<file-id>.

In the following, /srv/overleaf-s3-migration is used for storing the files in the new directory layout.

We can make use of tar for rewriting the layout:

mkdir -p /srv/overleaf-s3-migration/user_files \
         /srv/overleaf-s3-migration/template_files
docker exec sharelatex \
  tar --create --directory /var/lib/overleaf/data/user_files . \
| tar --extract --directory /srv/overleaf-s3-migration/user_files \
  --transform=sx_x/x
docker exec sharelatex \
  tar --create --directory /var/lib/overleaf/data/template_files . \
| tar --extract --directory /srv/overleaf-s3-migration/template_files \
  --transform=sx_x/xg

Step 2 upload the files

Depending on your preference, you can use the minio mc S3 client or the aws cli for uploading the files to your S3 compatible object storage.

aws cli

  • Here you should replace overleaf-user-files, overleaf-template-files, overleaf-project-blobs and overleaf-chunks with the names of your S3 buckets.

  • Also replace /srv/overleaf-bind-mount with the local path of the /var/lib/overleaf bind-mount. By default, this is ~/overleaf_data in a docker-compose.yml deployment and <toolkit-checkout>/data/overleaf when using the Toolkit.

aws s3 sync /srv/overleaf-s3-migration/user_files s3://overleaf-user-files
aws s3 sync /srv/overleaf-s3-migration/template_files s3://overleaf-template-files

aws s3 sync /srv/overleaf-bind-mount/data/history/overleaf-project-blobs s3://overleaf-project-blobs
aws s3 sync /srv/overleaf-bind-mount/data/history/overleaf-chunks s3://overleaf-chunks

minio mc

We are using the server alias "s3" here, you may have picked another name.

mc mirror /srv/overleaf-s3-migration/user_files s3/overleaf-user-files
mc mirror /srv/overleaf-s3-migration/template_files s3/overleaf-template-files

mc mirror /srv/overleaf-bind-mount/data/history/overleaf-project-blobs s3/overleaf-project-blobs
mc mirror /srv/overleaf-bind-mount/data/history/overleaf-chunks s3/overleaf-chunks

Step 3 start the instance pointing at S3

For Docker Compose deployments, you can also remove the bind-mount for the data directory from the volumes section.

You can now start the instance and validate the migration:

  • can preview binary files in the editor

  • can compile a PDF with images

  • can upload new files

Rolling back

You can roll back the migration gracefully in reversing the steps:

  1. Shutdown the instance

  2. Mirror back the files by flipping the sequence of source/destination

  3. Write new files back into the local directory using an inverse transform

  4. Restart the instance with the old configuration

# When using aws cli
aws s3 sync s3://overleaf-user-files /srv/overleaf-s3-migration/user_files
aws s3 sync s3://overleaf-template-files /srv/overleaf-s3-migration/template_files
aws s3 sync s3://overleaf-project-blobs /srv/overleaf-bind-mount/data/history/overleaf-project-blobs
aws s3 sync s3://overleaf-chunks /srv/overleaf-bind-mount/data/history/overleaf-chunks

# When using minio mc
mc mirror s3/overleaf-user-files /srv/overleaf-s3-migration/user_files
mc mirror s3/overleaf-template-files /srv/overleaf-s3-migration/template_files
mc mirror s3/overleaf-project-blobs /srv/overleaf-bind-mount/data/history/overleaf-project-blobs
mc mirror s3/overleaf-chunks /srv/overleaf-bind-mount/data/history/overleaf-chunks
# Write files into local Server CE/Server Pro
tar --create --directory /srv/overleaf-s3-migration/user_files . \
| docker exec --interactive sharelatex \
    tar \
      --extract \
      --keep-old-files \
      --directory /var/lib/overleaf/data/user_files \
      --transform=sx./xx --transform=sx/x_x \
      --wildcards '*/*/*'

tar --create --directory /srv/overleaf-s3-migration/template_files . \
| docker exec --interactive sharelatex \
    tar \
      --extract \
      --keep-old-files \
      --directory /var/lib/overleaf/data/template_files \
      --transform=sx./xx --transform=sx/x_xg \
      --wildcards '*/*/*/*/pdf-converted-cache/*' \
      --wildcards '*/*/*/*/pdf' \
      --wildcards '*/*/*/*/zip'

The first transform removes the top level folder. The 2nd transform changes the directory layout to a flat one. The wildcards ensure that only files are extracted, not their parent (project) folders.

Add all the S3 related variables to your config, as detailed in the section in the setup guide.

Please keep the in place.

support@overleaf.com
S3 Setup
S3 setup
bind-mount of a scratch disk for ephemeral files
S3
Overview of variables