Git Submodule with Local Changes Example

Ross Wolin - last updated 2015.11.18

---

The Problem

I am writing firmware for an embedded system which uses the uGFX graphics library. I use Git for source control, and wanted to use a Git submodule to embed uGFX's Bitbucket hosted repository into my source repository. Additionally, I wanted to be able to add some custom board and driver files to uGFX which are needed for my project, but are of no value to uGFX (and won't be pushed back to their repository.)

Initially, I merely added a Git submodule to my local repository, made some changes and commited them to the submodule's master branch. This worked smashingly... until I tried to make a local clone of my repository, using the recursive switch:

git clone --recursive myrepo myrepo_clone

and got an error:

fatal: reference is not a tree: ca025c09ca2b7b82ca086309eeb0696d674cb1d0 Unable to checkout 'ca025c09ca2b7b82ca086309eeb0696d674cb1d0' in submodule path 'ugfx_submodule'

the problem is that my changes to the submodule's master branch are in my local repository, but when Git tries to clone the uGFX submodule, it gets source from the Bitbucket repository, which obviously does not have my changes.

What I'm doing seems to me like a pretty common situation/workflow... surprisingsly, when I searched, I did not find documentation/examples about how to set this up properly so that recursive clones would work, etc.

Here's my requirements in more detail:

1. I need to get code from thce uGFX Bitbucket repository's master branch, pull in changes from that branch when available, and integrate them into my product after testing.
2. I need to be able to add code to the local uGFX tree and keep these additions under source control, but not push them back to the uGFX repository (my code is specific to my project and proprietary h/w, trust me, uGFX doesn't want it. =)
3. Other users must be able to clone my repository. They don't have to be able to update the uGFX tree with the Bitbucket changes (I will handle when those changes are integrated) --- although they could if they made minor modifications to their repository.

---

My Solution

I came up with a workable solution, involving a version of Triangular Workflow, using an upstream repository. I used a Raspberry Pi + Raspbian for the upstream repository because I had one on my network running DNS/DHCP and thought it would be cool, you could just as easily use a second local repository on your disk, Bitbucket/Github, etc, etc

Here are the specifics for my solution:

• Create an upstream Git server. I used a Raspberry Pi and put the repos on a USB stick, because it was convenient, starting with this article. I added some enhancements from the Git book, like adding a git user and some authorized public ssh keys, etc...


• Make two bare repositories on the Git server, for this example call them myproduct.git and myproduct_ugfx.git


• Push my repository's master branch to the myproduct.git repository. I didn't write this part down, but I believe I just added a remote for origin and pushed the master branch:

git remote add origin git@pi-b:/mnt/git_repos/myproduct.git
git push

• From the top directory of my repository, I add the uGFX submodule

git submodule add https://bitbucket.org/Tectu/ugfx.git ugfx_submodule

• cd into the submodule's directory, and change the name of the submodule's master branch to updates

git branch -m master updates

• Rename the remote origin to bitbucket, which the updates (formerly master) branch tracks

git remote rename origin bitbucket

• At this point, run git branch -vv as a check, and it should show one branch named updates which tracks remote bitbucket's master branch


• The submodule no longer has a master branch or origin. I wanted the submodule master branch to track my Pi Git server's uGFX repository, so I add a remote for origin, create a master branch, and push the master branch to the server (telling it to track that branch):

git remote add origin git@pi-b:/mnt/git_repos/myproduct_ugfx.git
git checkout -b master 
git push -u origin master

• Now that I've pushed the submodule to my server, I change the uGFX submodule to point to my server, so when people clone the whole repo with --recursive, they get my master branch with my additions

git config remote.origin.url git@pi-b:/mnt/git_repos/myproduct_ugfx.git

At this point, I wanted to add a few files to the uGFX tree. I add them to the master branch, commit, and push (which goes to the Pi repository.) Remember to also go up a direcotry, outside of the submodule, run git submodule sync (not sure if that's necessary), and then commit+push those changes to the main/outer repo.

Now, when there are changes on the Bitbucket repository, in the submodule I can pull them into my local updates branch, then when I've tested them and want to integrate them into the my product, I can merge updates with master, then commit and push master to the Pi repository.

Local recursive clones (like the one above that caused the initial error) will now work. If someone wants to clone the myproduct repository from the Pi, provided they have access to the Pi (see Git book for adding an ssh key) they can do it with

git clone --recursive git@pi-b:/mnt/git_repos/myproduct.git

---

Links

• Triangular workflow example
• Simple RPi Git Server
• Git book, server chapter

---

Send comments, questions, money in large denominations, etc to eng at mysticengineering.com