Upgrade Alfresco 3.4 to 4.2 in production enviroment

While searching the internet for a complete solution to upgrade my alfresco 3.4d system to the newer version 4.2 I noticed that all guides provided in the alfresco forum as well as the official wiki are of this kind:

1. Backup database and content store
2. Install new System
3. Import Backup of database and content store.

As much as they wished it was that easy and straight forward it is NOT.

I very much like the Alfresco Software. But upgrading is a horrific experience.
In my case over 20 upgrade attempts have been conducted (using VM snapshots which saved days of time) before finding a working procedure.

Please read the whole article before starting any actions to perform the upgrade or you will end up doing stuff several times instead of accomplishing everything in one run.

Depending on the current state of development (the builds you use, which can differ dramatically from night to night) you may or may not encounter the following issues while upgrading:

Workflows not working after upgrade:

Alfresco changed their BPM Engine from jBPM to Activity Engine since the 4.0 release. This introduces the following errors assuming you are upgrading a 3.4 that is a production system and not a vanilla installation.

Workflows coming from WebQuickStart and RM module in 3.4 will NOT work if you upgrade to 4.2 so you will have to undeploy them BEFORE making your backups for the upgrade.

If you managed to upgrade your system somehow without undeploying the workflows you will notice that not only the mentioned workflows will be broken but the whole workflow system will be rendered nonfunctional throwing “Definitions not found” errors. This will prevent you from perfoming ANY actions on ANY workflow or definition such as starting new workflows, even listing workflows, stopping running, deploying new workflows.
So you will end up with an upgraded system still not usable for business.

To prevent this please do the following:

• Tell your users to abort/finish all assigned workflows before a given date you scheduled for the upgrade
• Abort all unfinished workflows after that date using the workflows console:
  

  alfresco/faces/jsp/admin/workflow-console.jsp issue “delete all workflows”

• Undeploy all workflow definitions related to WCM e.g. wcm wfsm (in fact you can undeploy all jBPM definitions as you just deleted all running workflows and because you will migrate to activity anyway)
• This steps will clean up your original installation for the upgrade.

JBPM Tables blocking the upgrade procedure:

Alfresco will try to update the old JBPM tables during the upgrade using some sql scripts. Unfortunately you may encounter some case sensitivity problems (because someone from the alfresco 3.4 team decided to make all JBPM tables uppercase in contrast to the rest of the database -.-). Assuming you have a standalone MySQL Server on your target machine (where you want to import your database backup from 3.4) you will have to set the following entries in my.cnf BEFORE creating the target Database:

[mysqld]
lower_case_table_names = 0

UPDATE 12/04/13 (based on additions by Ken M. Jameson):
If you consider installing the Bitnami version of Alfresco 4.2 you should be prepared for some issues while upgrading. e.g. Ken found that “lower_case_table_names = 1” is contained inside the script that actually runs Alfresco and it is overriding the above statement in “my.cnf”.

I had to remove the one inside the startup script that was included with the Bitnami Alfresco installer (I at first tried to get this to work with the Bitnami version of Alfresco 4.2.e simply because their Alfresco included MYSQL instead of POSTGRES. I thought that would be easier, but never got it to work)
Not a problem, once I installed MYSQL in Ubuntu (using “apt-get install”).
I recommend using the regular Alfresco installer. Go to Alfresco’s site and download the one you need. I used 4.2.e (final).

kmj@mahlan.com

Now you can create your target database and import your 3.4 source db backup.

After that apply the following patch to make JBPM Tables uppercase again as the export-import (using mysqldump) will make them lowercase.

Upgrade blocking when trying to upgrade activity tables:

If you start your new 4.2 system for the first time to upgrade, it will try to execute a lot of DB schema patches. One of them will be to create new Activity Process Engine tables and another (or several others) to update those tables to match the current Activity version.
For sooome reason the update part of those scripts will be executed BEFORE the creation of the tables. The upgrade will fail dramatically.

To get around this I first installed a vanilla version of 4.2 with no data, extracted the create statements for the vanilla Activity tables from the DB and packed them into a patch file that I apply to my freshly imported backup before starting the upgrade.

At this point we have the tables created but still have to set some values and tell the database that the faulty patch has already been applied automatically using another modified patch from the 4.2 installation.

Well, we are done. Activity will be upgrading find now. ;)

Records management module in 3.4 kills SORL and Lucene search system:

If you are using the records management module in your source 3.4 installation you have really bad luck. As far as I know there is no uninstallation procedure for this module and there is no new module available for 4.2.
When trying to upgrade to 4.2 having all the database entries left over from the the old RM module the new SOLR indexer introduced in 4.2 will almost certainly die trying to index the RM transactions. This will also happen to the old lucene indexer that you will have to use for the initial indexing procedure after upgrade.

If you installed the Records Management module with example data but never used it for production there is a workaround for you.

First delete all the example records management sites from the 3.4 system before backing up. This will delete most of the records related entries from the database but there are still some left over. To remove these left over entries you will have to use the following statement:

delete from alf_child_assoc where parent_node_id= ANY (
SELECT id from alf_node where type_qname_id= ANY (
select id from alf_qname where local_name LIKE ‘xxxxxx’));

Execute this statement replacing the “xxxxxx” with “record%” than again with “disposition%” and again with “filePlan”.

This will remove a lot of RM nodes so that SOLR and Lucene indexer will survive the initial indexing procedure.

Additionally add “lucene.indexer.batchSize=1000” to the alfresco-global.properties file of you new 4.2 installation before starting the upgrade to ease workload on lucene indexer.

Errors present after the upgrade:

After your managed to upgrade the system to 4.2 you may see reccuring errors in the system logs:

Failed to execute script ‘classpath*:alfresco/site-webscripts/org/alfresco/components/document-details/document-workflows.get.js’: 01120002 Error: Could not load document workflows

This may be caused by old workflow definitions. To stop the error from showing up again and again you could try adding
system.workflow.engine.jbpm.enabled=true
to alfresco-global.properties to additionally enable the old jbpm engine.

Another Error may be:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This is caused because you have installed the jvm bundled with your alfresco 4.2 release. Unfortunately the bundled jvm has expired certificates. So you will have to regenerate them. This may sound trivial but it is definitely not. Fortunately there is a bash script to do this. ;)

Enable UniCode support:

To enable UniCode support you will have to set the correct MySQL Server settings BEFORE importing your DB backup:

[mysqld]

init_connect=’SET collation_connection = utf8_general_ci’
collation-server = utf8_unicode_ci
init-connect=’SET NAMES utf8′
character-set-server = utf8

Actually, to make this work with the tomcat server you will also have to append
“?useUnicode=yes&characterEncoding=UTF8”
to your DB connection URI during initial 4.2 installation.
Further on edit the tomcat’s server.xml and add “URIEncoding="UTF-8"" to the connectors.

STEP BY STEP GUIDE (for the Linux savvy):

IMPORTANT:
I assume you have cloned your production machine to machine A. Do NOT modify the DB on the production machine until your new 4.2 installation has been sufficiently tested.

Machine A: source machine alfresco 3.4 MySQL

Machine B: target machine

1. A: Delete all workflows and undeploy all definitions. (see alfresco wiki)
   Also Delete all Records Management Sites you may have created.
2. A: Stop alfresco and start Mysql Server alone.
   If you use the bundled MySQL server you can find the command to start the bundled server in the alfresco.sh start script. Look for the mysqld_safe script and start it providing the path to the data directory of the alfresco db.
3. A: Connect to the DB with mysql-client and delete the left over RM nodes by issuing the following SQL:
   

   delete from alf_child_assoc where parent_node_id= ANY (SELECT id from alf_node where type_qname_id= ANY (select id from alf_qname where local_name LIKE ‘record%’));delete from alf_child_assoc where parent_node_id= ANY (SELECT id from alf_node where type_qname_id= ANY (select id from alf_qname where local_name LIKE ‘disposition%’));

   delete from alf_child_assoc where parent_node_id= ANY (SELECT id from alf_node where type_qname_id= ANY (select id from alf_qname where local_name LIKE ‘filePlan’));

   UPDATE 12/04/13 (based on additions by Ken M. Jameson):
   There are also some user and system groups inside Share that are related to Records_Management. You should remove them via the admin console from inside the system.

   Now you are ready to take the backup. Mysqldump your DB and move it to machine B.

4. B: Install mysql Server modify my.cfn with this settings and restart:
   

   [mysqld]
   lower_case_table_names = 0
   init_connect=’SET collation_connection = utf8_general_ci’
   collation-server = utf8_unicode_ci
   init-connect=’SET NAMES utf8′
   character-set-server = utf8

5. B: Create a new database.UPDATE 12/04/13 (based on additions by Ken M. Jameson):
   Before you can accomplish this, you should make sure to stop Alfresco (if it was already installed in a prior failed upgrade attempt) and start MYSQL

   server.service mysql status
   service mysql start
   mysql –u root –pyourpassword
   mysql> show databases;
   #optional if this is your second upgrade attempt: mysql> drop database alfresco;
   mysql> create database alfresco;
   mysql> quit;

6. B: Install prequistites for OpenOffice (example for Debian).

   apt-get install libfontconfig1-dev libsm-dev libice-dev libXrender-dev libXext-dev

7. B: Install Alfresco 4.2 using the same modules you have chosen for 3.4 (e.g. WebQuickStart). Do not install Postgres DB. DO install the bundled JVM.
   Use “org.gjt.mm.mysql.Driver” for DB Driver and

   “jdbc:mysql://localhost:3306/<DBNAMEHERE>?useUnicode=yes&characterEncoding=UTF8”

   for Database connection uri.

8. B: After installation is finished do NOT let the installer start Alfresco after it finished installtion.
9. B: Drop the created database from step 5 and create a new one with the same name.
   

   service mysql status
   service mysql start
   mysql -uroot -pyourpassword
   mysql> show databases;
   mysql> drop database alfresco;
   mysql> create database alfresco;
   mysql> quit;

   Now you can import your DB Backup from step 3.

   mysql -uroot -pyourpassword alfresco < /path_to_your_backup_from_3_4/olddata.sql

10. B: After import is finished apply uppercase-JBPM-Tables-patch.UPDATE 12/04/13 (based on additions by Ken M. Jameson):
    To verify that patching is performed correctly Ken did this before applying the patch:

    mysql –uroot –ppassword
    mysql> show databases;
    mysql> use alfresco;
    mysql> show tables;
    Total of 88 tables
    #JBPMN tables should all be lowercase due to dump_import
    mysql> quit;

    Now apply the Patch that will make tables uppercase:

    mysql -uroot –pyourpassword alfresco</path_to_patch/uppercase-JBPM-Tables-patch.txt

    #verify that the patch worked
    mysql –uroot –ppassword
    mysql> show databases;
    mysql> use alfresco;
    mysql> show tables;
    #Verify that JBPMN tables are now uppercase … should be at the top of list
    Total tables should still be 88 …
    mysql> quit;

11. B: Apply create-Activity-Tables-patch.UPDATE 12/04/13 (based on additions by Ken M. Jameson):
    While applying this patch Ken found that you have to disable foreign key checks to make it work. I think it depends on your mysql-server version if you need it or not.

    #I had to add to this “patch” …
    SET FOREIGN_KEY_CHECKS = 0  as first line in “patch”
    SET FOREIGN_KEY_CHECKS = 1 as last line in “patch”
    #Before running this patch, I did …
    mysql –uroot –ppassword
    mysql> show databases;
    mysql> use alfresco;
    mysql> show tables;
    Total of 88 tables …
    #Now exit mysql …
    mysql> quit;
    #and run patch …
    mysql -uroot –pyourpassword alfresco <
    /opt/alfresco_activity_table_creation_script.txt
    #After running this patch, I did …
    mysql –uroot –ppassword
    mysql> show databases;
    mysql> use alfresco;
    mysql> show tables;
    #Verify that ACT_??? tables have been added to the top of the list …
    #Total tables should be 109 …
    #Now exit mysql …
    mysql> quit;

12. B: Apply confirm-Activity-Tables-creation-patch.
    

    mysql -uroot –pyourpassword alfresco </path_to_confirm_Activity_tables_creation_patch/confirm-Acrivity-TAbles-creation-patch.txt

13. B: Download MySQL Connector/J and place it into <alfrescoInstallDir>/tomcat/lib.
14. B: Add the Property URIEncoding="UTF-8"  to the connectors in tomcat’s server.xml
15. B: Edit <alfrescoInstallDir>/tomcat/shared/classes/alfresco-global.properties:
    

    #not sure which one is implemented in the current version. Just use both.
    schema.update=true
    db.schema.update=true
    #COMMENT  OUT ALL EMAIL SETTINGS FOR THE FIRST START#Comment out Solr and enable Lucene with full reindex#index.subsystem.name=solr
    dir.keystore=${dir.root}/keystore
    solr.port.ssl=8443
    index.subsystem.name=lucene
    index.recovery.mode=FULL
    lucene.indexer.batchSize=1000#Disable IMAP as long it is not configured
    imap.server.enabled=false#I enable JOD Converter because it works better for me.
    jodconverter.enabled=true#Enable JBPM Engine for the upgrade
    system.workflow.engine.jbpm.enabled=true

16. A: Copy all data from alf_data to Machine B/<AlfrescoInstallDir>/alf_data.
17. B: Delete alf_data/lucene-indexes. Rename lucene-indexes-backup to lucene-indexes
18. Execute regenerate-certificates-script to renew jvm certificates.UPDATE 12/04/13 (based on additions by Ken M. Jameson):
    Ken found that the recent downloadable installers already contain the renewed certificates.
    So this step is not necessary anymore.
19. You are ready to start the upgrade. If you work in a Virtualized Enviroment I would recommend to snapshot Machine B now. So you can rollback and change something if you (or I ;) forgot to change some settings.Execute “/<alfrescoInstallDir>/alfresco.sh start” and watch the upgrade progress with “tail -f /<alfrescoInstallDir/tomcat/logs/catalina.out”

During the first boot alresco should patch your Database Scheme. There are some operations that can take very long. e.g. ACL Operations. Don’t despair. Depending on your DB it can take from minutes to hours.
Further on when Lucene indexing kicks in you will receive a lot of error about nodes and transactions not beeing parsed because we deleted RM nodes. This is ok. Just ignore the errors and let Lucene complete.

If you are sure that your machine hungup and not processing the reindexing anymore, “killall java” tomcat, delete tomcat/temp/catalina.pid and start alfresco once again. The indexing will beginn again and hopefully pass this time.

After your new System is up and running do NOT switch off lucene in alfresco-global.properties at least for a few days (depending on your DB and content store size).
SOLR will need some time to catch up with indexing. The search system will not work properly until SOLR indexed all transactions ever made.

Watch catalina.log for errors and have fun with your finally upgraded Alfresco 4.2 System.

Comments are welcome. ;)

UPDATE 12/04/13:

Some Additional Points:

• There are some issues with the activity feed notifier if you upgraded your install. It mainly will try to send out emails to users about the recent site activities and cause some errors if the users subscribed to the feed but you did not configure emailing by now. To prevent this just add “activities.feed.notifier.enabled=false” to your alfresco-global.properties file.
• There also might be issues with preview generation of some filetypes. Add the following to alfresco-global.properties (if not already there) to improve preview generation with jodconverter.“

  jodconverter.enabled=true
  jodconverter.officeHome=/pathtoalfresco/libreoffice
  jodconverter.portNumbers=8100″

• Strange thing is that by default the 4.2 will not send out site-invite-emails. To enable this you should also add “notification.email.siteinvite=true” to alfresco.global.properties.
• If you are much into photography some of your pictures may be way to big for the image-transformers to convert if you upload like 100 of them at once. Your server will be 100% busy for hours and the converters may die while trying to generate thumbnails and previews. ( not without hogging all your RAM before dying ;)
  Add alfresco_4.2_thumbnailing_setup.txt to your alfresco-global.properties and set reasonable size limits for thumbnail generation. (better no thumbnail than an unusable slow system)