Full project history migration
Last updated
Was this helpful?
Last updated
Was this helpful?
The 3.5.x release of the Community Edition includes the that is already available in our SaaS offering,
After upgrading your instance to Overleaf CE 3.5.13, all new projects will be using Full Project History by default. Existing projects will continue using the legacy History system, until they’re migrated.
The new Full Project History brings several improvements for users:
It tracks changes in binary files, which is unsupported in the legacy system.
There is support for labelled versions.
The system is in general more robust, there is less chance of data loss.
Check for more information about full project history.
--force-clean clears partially migrated project history data in the new system, this allows retrying the migration for individual projects that failed in prior attempts;
--fix-invalid-characters replaces non printable characters that are not supported by the new history system;
--convert-large-docs-to-file converts documents that are above the 2MB editable-size threshold to a non-editable file)
The output should look like this:
If the migration is successful, you'll get an exit code of 0, and the last lines indicating no failures:
You can reopen access to your users (see next step). If there are failures, please see the troubleshooting section below. You can still reopen the site if the problems are not immediately fixed, and the unmigrated projects will remain on the legacy history system.
If you had chosen to perform an offline migration then you will need to re-open the site. If you are still logged in you will need to:
Click on the Admin button and choose Manage Site
Click the Open/Close Editor tab
Click the Reopen Editor button
If you have closed your browser then you will need to restart the site with $ bin/up.
To prevent users from being able to log in while the history migration script is running, please follow these steps:
Log into your Overleaf instance with an admin account
Click on the Admin button and choose Manage Site
Click the Open/Close Editor tab
Click on the Close Editor button
Click on the Disconnect all users button
Once this has been done, if any users are logged in they will get redirected to the maintenance page, and any new users visiting the log-in page will see the maintenance page and won't be able to log in.
It is possible to run the migration scripts while the application is still running. There are a few considerations to take into account:
The migration process is CPU intensive, you should monitor resource usage while the script is running.
With a high --concurrency value, the event loop in some services (track-changes in particular) might experience some blocking, which would lead to a degraded UX experience. We recommend starting with the default --concurrency=1 value.
You can stop the script at any time. Starting it again will resume the migration when you left it. This is useful in case you prefer to run the migration in less busy hours (e.g. at night).
Our recommendation is to close the site and run the migration offline in a maintenance window when your project count is less than 1000 projects (db.projects.count()). If the number of projects is large you can run the script and monitor its progress, then decide whether to continue running it online or offline based on your particular case.
A script to cleanup legacy history data was added in Server Pro 3.5.6, 4.0.6 and 4.1.0.
The script can be run after all the projects have been migrated. It can also be used to free some space while performing an online migration.
We will add troubleshooting advice here. Please note that while we normally offer support only to Server Pro customers, given the nature of this migration, we will also do our best to support CE customers who experience problems specific to the full project history migration.
Subject: Full project history migration problem
Instance Type: CE or Server Pro (delete as appropriate)
Installation Type: Overleaf toolkit or docker-compose.yml or other (delete as appropriate)
Version: 3.5.x (toolkit: $ cat config/version)
Migration script output (which should be located in the container under /overleaf/services/web)
Migrated Projects: (as per migration script output)
Total Projects: (as per migration script output)
Remaining Projects: (as per migration script output)
Duration of the migration:
bin/doctor output (when using toolkit)
Toolkit version: $ git rev-parse HEAD (when using Toolkit)
Consider attaching the log files for the history-v1, project-history and track-changes services to the email. You can find these at /var/log/sharelatex inside the sharelatex container and export them like this:
Please redact any sensitive information from the log files before attaching them.
The migration may fail for projects which have a malformed file tree (for example, where filenames are empty). You can find a list of these problems using the find_malformed_filetrees script which checks all projects in the database:
To fix the invalid paths, use the fix_malformed_filetree script, running the command once for each bad path:
If there is a project that has been migrated to full project history but you want to go back to the legacy history, use the downgrade_project script as follows:
If the full project history migration script fails (i.e. exits with an error or prints a non-zero number of failed projects), please send the following details to our support team by email , detailing: