An SDLC for Sugar

sugarcrmdevelopers —  September 16, 2011 — 13 Comments

Editor’s Note: Please check out the note from Heather after the attachment on a update to this document.

At SugarCon this past spring, Heather Johnson from the Robin Hood Foundation did an excellent talk about her work in developing a Systems Development Life Cycle ( or SDLC for short ) for the SugarCRM instance in use by the group. It is a very customized instance with lots of moving parts, and with the limited documentation and use cases out there for such an endeavor, she put together a guide detailing the strategy they used to manage it. It’s a great document and provides a concise and complete strategy to properly manage software development on the Sugar Platform.

Download An SDLC for Sugar

Added 11/1/2011
A quick update on this that might be helpful: We have found that, using the method described in the article above, we often have to commit changes that result from running a Quick Repair and Rebuild. Not only is this annoying but it can potentially create merge conflicts. For example, running a QR&RB in Production may modify files that have to be merged backwards in the lifecycle to SUGARDEV and though one assumes that the same changes would have been made to the same files when one ran QR&RB in SUGARDEV a few minutes earlier, the world just doesn’t always seem to work that way – and who wants to deal with the overhead of pulling down QR&RB-related changes anyway? So we have made the following change in policy: In addition to ignoring the cache, we now ignore all files that are automatically generated by a QR&RB. No such files are tracked in the index. Since many of these auto-generated files are critical to Sugar’s operation, the upshot of this change in policy is that you now *have* to run a QR&RB each time you promote changes up through the lifecycle that would require it – not just in your sandbox. This has the effect of taking the changes made to tracked files, and using them to take care of any modifications that are needed to the auto-generated files. Since no auto-generated files are tracked in the index, there is no more worry about seeing such files show up, unwanted, in a git status. Here is a quick how-to on getting rid of these auto-generated files from your repo (assuming you started out by tracking them and that you are doing something very like what is described in the attached article):

On SUGARDEV and on your sandbox, you should replace your current .gitignore file with the following:

../*.log
../cache/
../config.php
../config_override.php
../custom/themes/default/images/company_logo.png
../custom/application/*
../custom/modules/*/Ext/*

(Note, we don’t track company_logo.png because we like to make cute little logos for the different environments we’re developing in)

Still on SUGARDEV:
1. Remove the unwanted files from your index (but leave them intact in the working directory):

git rm -r --cached custom/application/* custom/modules/*/Ext/*

2. Update any branches off the main branch on DEV. First switch to the branch you want to update

git checkout //note, you may have to do a git checkout -f to force the checkout to occur as git sees that by switching branches it will be overwriting some working tree files no longer indexed in the current branch with files that are still indexed in yours. This should only be an issue once.

3. Then merge the MASTER branch into yours – this will remove all of the auto-generated files from the index associated with your branch
git merge MASTER

4. For any other active branches on SUGARDEV, repeat steps 2 & 3.

5. Re-run your permissions

sudo chown -R www-data.www-data /sugar/doc/root
sudo chmod -R ug+rw /sugar/doc/root

on your SANDBOX:

6. Pull down the changes. On your sandbox, do:

git pull origin

After you do this, you should be able to run a QR&RB any time you want to in any environment and it will not modify the index or create files that need checking in. Once again, only files that are looked at as the *source* of the auto-generated files will be indexed.

Thank you to Steve Williams of AtCore Systems for this suggested change!

13 responses to An SDLC for Sugar

  1. 

    Really enjoyed this presentation and I’m glad to see it’s been pointed out here.

    As much as I love a 400+char regex (this one is a true beast of regex and grep), I feel like it could have been narrowed down quicker/easier by just searching for specific queries (INSERT into fields_meta_data…, ALTER…, and CREATE…, DELETE, DROP). Am I missing one or several types of query that could be created by Studio or elsewhere in the system?

    • 

      Hi Matthew,

      Thanks for your comment! You are definitely right that it should be possible to write a cleaner regex than this. Your approach, at a high level, amounts to starting with nothing and telling the log specifically what to include. My approach was the opposite: start with everything and tell the log what the exclude. I did this primarily because I wasn’t confident that I knew exactly what I wanted from the log each time the db was modified and didn’t want to miss anything. Do you have an alternative that would make it faster/easier without losing any of the good stuff?

      • 

        I figured the same, that our approaches were coming from different angles. Here’s the search I’ve been using directly into my IDE:

        (?<=QUERY:)((INSERT into fields_meta_data)|(UPDATE fields_meta_data)|(INSERT into config)|(UPDATE config)|(alter table)|(create table)|(drop table)).+

      • 

        Restructured more and included several tables that were previously skipped over – perhaps grabbing all queries and then removing the excess was a better path. My search doesn’t yet attempt to touch workflow-related tables.

        (?<=QUERY:)(((INSERT into )|(UPDATE )|(DELETE from ))((fields_meta_data)|(config)|(acl_actions)|(acl_roles_actions)|(acl_fields)|(acl_roles)|(acl_roles_users)|(users)|(teams)|(team_memberships)).+)|((((alter)|(create)|(drop)) table)).+

  2. 

    A quick update on this that might be helpful: We have found that, using the method described in the article above, we often have to commit changes that result from running a Quick Repair and Rebuild. Not only is this annoying but it can potentially create merge conflicts. For example, running a QR&RB in Production may modify files that have to be merged backwards in the lifecycle to SUGARDEV and though one assumes that the same changes would have been made to the same files when one ran QR&RB in SUGARDEV a few minutes earlier, the world just doesn’t always seem to work that way – and who wants to deal with the overhead of pulling down QR&RB-related changes anyway? So we have made the following change in policy: In addition to ignoring the cache, we now ignore all files that are automatically generated by a QR&RB. No such files are tracked in the index. Since many of these auto-generated files are critical to Sugar’s operation, the upshot of this change in policy is that you now *have* to run a QR&RB each time you promote changes up through the lifecycle that would require it – not just in your sandbox. This has the effect of taking the changes made to tracked files, and using them to take care of any modifications that are needed to the auto-generated files. Since no auto-generated files are tracked in the index, there is no more worry about seeing such files show up, unwanted, in a git status. Here is a quick how-to on getting rid of these auto-generated files from your repo (assuming you started out by tracking them and that you are doing something very like what is described in the attached article):

    On SUGARDEV and on your sandbox, you should replace your current .gitignore file with the following:

    ../*.log

    ../cache/

    ../config.php

    ../config_override.php

    ../custom/themes/default/images/company_logo.png

    ../custom/application/*

    ../custom/modules/*/Ext/*

    (Note, we don’t track company_logo.png because we like to make cute little logos for the different environments we’re developing in)

    Still on SUGARDEV:

    1. Remove the unwanted files from your index (but leave them intact in the working directory):
        git rm -r –cached custom/application/* custom/modules/*/Ext/*

    2. Update any branches off the main branch on DEV. First switch to the branch you want to update

        git checkout         //note,
    you may have to do a git checkout -f to force the checkout to
    occur as git sees that by switching branches it will be overwriting
    some working tree files no longer
    indexed in the current branch with files that are still indexed in
    yours. This should only be an issue once.

    3. <!–
    @page { margin: 0.79in }
    P { margin-bottom: Then merge
    the MASTER branch into yours – this will remove all of the
    auto-generated files from the index associated with your branch

        git merge MASTER 

    4. For any other active branches on SUGARDEV, repeat steps 2 & 3.  

    5. Re-run your permissions
        sudo chown -R www-data.www-data /sugar/doc/root
        sudo chmod -R ug+rw /sugar/doc/root

    on your SANDBOX:

    6. Pull down the changes. On your sandbox, do:

        git pull origin                   

    After you do this, you should be able to run a QR&RB any time you
    want to in any environment and it will not modify the index or create
    files that need checking in. Once again, only files that are looked at as the
    *source* of the auto-generated files will be indexed.

    Thank you to Steve Williams of AtCore Systems for this suggested change!

    • 

      AFAIK all (is this really so?) Sugar-generated files has ‘.ext.php’ extension. So the last two lines of .gitignore given above IMO could be replaced with just a ‘*.ext.php’ line.

Trackbacks and Pingbacks:

  1. What is the proper way to version control customization and setup a dev enrionment - SugarCRM Forums - October 11, 2011

    […] […]

  2. CVS custom only or from root? - March 27, 2012

    […] […]

  3. Copying from Development to Production - June 19, 2012

    […] […]

  4. SugarCRM SDLC – Monitoring Database Changes - August 29, 2012

    […] talking about the SugarCRM Software Development Life Cycle (SDLC), the bane of the experience is always the process of analyzing database changes and porting them to […]

  5. Using CVS for deployment and creating new fields. What's the best way? - August 30, 2012

    […] […]

  6. SugarCRM Developer Blog » Blog Archive » Best practices for backing up Sugar instances - January 24, 2013

    […] to the codebase very infrequently, and leveraging a VCS system is a good tool to use here ( see this blog post and this one for some ideas SDLC policies for Sugar […]

  7. best way to build/deploy new modules, and version control. - January 30, 2013

    […] […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s