githubFork/gitblit.git - Solinfo Gitblit

James Moger

2011-12-22 e6935876b97a63bae2ec087b4fc390c832aef155

commit \| author \| age
831469	1	## Federating Gitblit
JM	2
	3	SINCE 0.6.0
	4
ea936a	5	A Gitblit federation is a mechanism to clone repositories and keep them in sync from one Gitblit instance to another. Federation can be used to maintain a mirror of your Gitblit instance, to aggregate repositories from developer workstations, or to initially clone groups of repositories to developer workstations. If you are/were a Subversion user you might think of this as [svn-sync](http://svnbook.red-bean.com/en/1.5/svn.ref.svnsync.html), but better.
831469	6
d7fb20	7	If your Gitblit instance allows federation and it is properly registered with another Gitblit instance, each of the non-excluded repositories of your Gitblit instance can be mirrored, in their entirety, to the pulling Gitblit instance. You may optionally allow pulling of user accounts and backup of server settings.
JM	8
	9	The federation feature should be considered a security backdoor and enabled or disabled as appropriate for your installation.<br/>
	10	Please review all the documentation to understand how it works and its limitations.
ea936a	11
JM	12	![block diagram](fed_aggregation.png "Gitblit Federation Aggregation")
831469	13
d4c908	14	### Important Changes to Note
JM	15
997c16	16	The Gitblit 0.8.0 federation protocol adds retrieval of team definitions. Older clients will not know to request team information.
JM	17
	18	The Gitblit 0.7.0 federation protocol is incompatible with the 0.6.0 federation protocol because of a change in the way timestamps are formatted.
d4c908	19
JM	20	Gitblit 0.6.0 uses the default [google-gson](http://google-gson.googlecode.com) timestamp serializer which generates locally formatted timestamps. Unfortunately, this creates problems for distributed repositories and distributed developers. Gitblit 0.7.0 corrects this error by serializing dates to the [iso8601](http://en.wikipedia.org/wiki/ISO_8601) standard. As a result 0.7.0 is not compatible with 0.6.0. A partial backwards-compatibility fallback was considered but it would only work one direction and since the federation mechanism is bidirectional it was not implemented.
	21
2548a7	22	### Origin Gitblit Instance Requirements
831469	23
JM	24	- git.enableGitServlet must be true, all Git clone and pull requests are handled through Gitblit's JGit servlet
2c32fd	25	- federation.passphrase must be non-empty
2548a7	26	- The Gitblit origin instance must be http/https accessible by the pulling Gitblit instance.<br/>That may require configuring port-forwarding on your router and/or opening ports on your firewall.
831469	27
2c32fd	28	#### federation.passphrase
831469	29
2c32fd	30	The passphrase is used to generate permission tokens that can be shared with other Gitblit instances.
831469	31
2c32fd	32	The passphrase value never needs to be shared, although if you give another Gitblit instance the ALL federation token then your passphrase will be transferred/backed-up along with all other server settings.
831469	33
2c32fd	34	This value can be anything you want: an integer, a sentence, an haiku, etc. You should probably keep the passphrase simple and use standard Latin characters to prevent Java properties file encoding errors. The tokens generated from this value are affected by case, so consider this value CASE-SENSITIVE.
831469	35
2c32fd	36	The federation feature is completely disabled if your passphrase value is empty.
831469	37
3b5289	38	NOTE:
2c32fd	39	Changing your federation.passphrase will break any registrations you have established with other Gitblit instances.
831469	40
JM	41	### Pulling Gitblit Instance Requirements
	42
2548a7	43	- consider setting federation.allowProposals=true to facilitate the registration process from origin Gitblit instances
831469	44	- properly registered Gitblit instance including, at a minimum, url, federation token, and update frequency
JM	45
	46	### Controlling What Gets Pulled
	47
2548a7	48	If you want your repositories (and optionally users accounts and settings) to be pulled by another Gitblit instance, you need to register your origin Gitblit instance with a pulling Gitblit instance by providing the url of your Gitblit instance and a federation token.
831469	49
f6740d	50	Gitblit generates the following standard federation tokens:
831469	51	%BEGINCODE%
2c32fd	52	String allToken = SHA1(passphrase + "-ALL");
JM	53	String usersAndRepositoriesToken = SHA1(passphrase + "-USERS_AND_REPOSITORIES");
	54	String repositoriesToken = SHA1(passphrase + "-REPOSITORIES");
831469	55	%ENDCODE%
JM	56
3b5289	57	The ALL token allows another Gitblit instance to pull all your repositories, user accounts, and server settings.
JM	58	The USERS_AND_REPOSITORIES token allows another Gitblit instance to pull all your repositories and user accounts.
831469	59	The REPOSITORIES token only allows pulling of the repositories.
JM	60
	61	Individual Gitblit repository configurations such as description and accessRestriction are always mirrored.
	62
2c32fd	63	If federation.passphrase has a non-empty value, the federation tokens are displayed in the log file and are visible, to administrators, in the web ui.
831469	64
f6740d	65	The three standard tokens grant access to ALL your non-excluded repositories. However, if you only want to specify different groups of repositories to be federated then you need to define federation sets.
JM	66
8f73a7	67	#### Federation Sets
JM	68
f6740d	69	Federation Sets (federation.sets) are named groups of repositories. The Federation Sets are defined in `gitblit.properties` and are available for selection in the repository settings page. You can assign a repository to one or more sets and then distribute the federation token for the set. This allows you to grant federation pull access to a subset of your available repositories. Tokens for federation sets only grant pull access for the member repositories.
831469	70
2548a7	71	### Federation Proposals (Origin Gitblit Instance)
831469	72
2c32fd	73	Once you have properly setup your passphrase and can see your federation tokens, you are ready to share them with a pulling Gitblit instance.
831469	74
3b5289	75	The registration process can be partially automated by sending a federation proposal to the pulling Gitblit instance.
831469	76	To send a proposal:
JM	77
	78	1. Login to your Gitblit instance as an administrator
3b5289	79	2. Select and click the propose link for the appropriate token on the federation page
d7fb20	80	3. Confirm the publicly accessible url of your (origin) Gitblit instance
JM	81	4. Enter the url of the pulling Gitblit instance you want to federate with
	82	5. Optionally enter a message for the administrators
	83	6. Click propose
831469	84
JM	85	Not all Gitblit instances accept federation proposals, there is a setting which allows Gitblit to outright reject them. In this case an email or instant message to the administrator of the other Gitblit instance is required. :)
	86
d7fb20	87	If your proposal is accepted, the proposal is cached to disk on the pulling Gitblit server and, if properly configured, the administrators of that Gitblit server will receive an email notification of your proposal.
831469	88
JM	89	Your proposal includes:
	90
	91	1. the url of your Gitblit instance
	92	2. the federation token you selected and its type
	93	3. the list of your non-excluded repositories, and their configuration details, that you propose to share
	94
3b5289	95	Submitting a proposal does not automatically register your server with the pulling Gitblit instance.
831469	96	Registration is a manual process for an administrator.
JM	97
	98	### Federation Proposals (Pulling Gitblit Instance)
	99
	100	If your Giblit instance has received a federation proposal, you will be alerted to that information the next time you login to Gitblit as an administrator.
	101
d7fb20	102	You may view the details of a proposal by scrolling down to the bottom of the repositories page and selecting a proposal. Sample registration settings will be generated for you that you may copy & paste into either your `gitblit.properties` file or your `web.xml` file.
831469	103
2548a7	104	### Excluding Repositories (Origin Gitblit Instance)
831469	105
f6740d	106	You may exclude a repository from being pulled by any federated Gitblit instance by setting its federation strategy to EXCLUDE in the repository's settings page.
831469	107
JM	108	### Excluding Repositories (Pulling Gitblit Instance)
	109
	110	You may exclude repositories to pull in a federation registration. You may exclude all or you may exclude based on a simple fuzzy pattern. Only one wildcard character may be used within each pattern. Patterns are space-separated within the exclude and include fields.
	111
	112	federation.example.exclude = skipit.git
	113
	114	OR
	115
	116	federation.example.exclude = *
	117	federation.example.include = somerepo.git someotherrepo.git
	118
	119	OR
	120
	121	federation.example.exclude = *
	122	federation.example.include = common/* library/*
	123
	124	### Tracking Status (Pulling Gitblit Instance)
	125
2548a7	126	Below the repositories list on the repositories page you will find a section named federation registrations. This section enumerates the other gitblit servers you have configured to periodically pull. The status of the latest pull will be indicated on the left with a colored circle, similar to the status of an executed unit test or Hudson/Jenkins build. You can drill into the details of the registration to view the status of the last pull from each repository available from that origin Gitblit instance. Additionally, you can specify the federation.N.notifyOnError=true flag, to be alerted via email of regressive status changes to individual registrations.
831469	127
2548a7	128	### Tracking Status (Origin Gitblit Instance)
831469	129
2548a7	130	Origin Gitblit instances can not directly track the success or failure status of Pulling Gitblit instances. However, the Pulling Gitblit instance may elect to send a status acknowledgment (federation.N.sendStatus=true) to the origin Gitblit server that indicates the per-repository status of the pull operation. This is the same data that is displayed on the Pulling Gitblit instances ui.
831469	131
2548a7	132	### How does it work? (Origin Gitblit Instances)
831469	133
JM	134	A pulling Gitblit instance will periodically contact your Gitblit instance and will provide the token as proof that you have granted it federation access. Your Gitblit instance will decide, based on the supplied token, if the requested data should be returned to the pulling Gitblit instance. Gitblit data (user accounts, repository metadata, and server settings) are serialized as [JSON](http://json.org) using [google-gson](http://google-gson.googlecode.com) and returned to the pulling Gitblit instance. Standard Git clone and pull operations are used to transfer commits.
	135
	136	The federation process executes using an internal administrator account, $gitblit. All the normal authentication and authorization processes are used for federation requests. For example, Git commands are authenticated as $gitblit / token.
	137
2c32fd	138	While the $gitblit account has access to all repositories, server settings, and user accounts, it is prohibited from accessing the web ui and it is disabled if federation.passphrase is empty.
831469	139
JM	140	### How does it work? (Pulling Gitblit Instances)
	141
	142	Federated repositories defined in `gitblit.properties` are checked after Gitblit has been running for 1 minute. The next registration check is scheduled at the completion of the current registration check based on the registration's specified frequency.
	143
	144	- The shortest frequency allowed is every 5 minutes
	145	- Decimal frequency values are cast to integers
	146	- Frequency values may be specified in mins, hours, or days
	147	- Values that can not be parsed default to 60 minutes
	148
	149	After a repository has been cloned it is flagged as isFederated (which identifies it as being sourced from another Gitblit instance), isFrozen (which prevents Git pushes to this mirror) and federationStrategy=EXCLUDED (which prevents this repository from being pulled by another federated Gitblit instance).
	150
	151	#### Origin Verification
	152
3b5289	153	During a federated pull operation, Gitblit does check that the origin of the local repository starts with the url of the federation registration.
831469	154	If they do not match, the repository is skipped and this is indicated in the log.
JM	155
997c16	156	#### User Accounts & Teams
831469	157
997c16	158	By default all user accounts and teams (except the admin account) are automatically pulled when using the ALL token or the USERS_AND_REPOSITORIES token. You may exclude a user account from being pulled by a federated Gitblit instance by checking exclude from federation in the edit user page.
831469	159
93f472	160	The pulling Gitblit instance will store a registration-specific `users.conf` file for the pulled user accounts and their repository permissions. This file is stored in the federation.N.folder folder.
831469	161
997c16	162	If you specify federation.N.mergeAccounts=true, then the user accounts and team definitions from the origin Gitblit instance will be integrated into the `users.conf` file of your Gitblit instance and allow sign-on of those users.
831469	163
3b5289	164	NOTE:
JM	165	Upgrades from older Gitblit versions will not have the #notfederated role assigned to the admin account. Without that role, your admin account WILL be transferred with an ALL or USERS_AND_REPOSITORIES token.
d7fb20	166	Please consider adding the #notfederated role to your admin account!
831469	167
JM	168	#### Server Settings
	169
d7fb20	170	Server settings are only pulled when using the ALL token.
831469	171
JM	172	The pulling Gitblit instance will store a registration-specific `gitblit.properties` file for all pulled settings. This file is stored in the federation.N.folder folder.
	173
	174	These settings are unused by the pulling Gitblit instance.
	175
	176	### Collisions and Conflict Resolution
	177
	178	Gitblit does not detect conflict and it does not offer conflict resolution of repositories, users, or settings.
	179
	180	If an object exists locally that has the same name as the remote object, it is assumed they are the same and the contents of the remote object are merged into the local object. If you can not guarantee that this is the case, then you should not store any federated repositories directly in git.repositoriesFolder and you should not enable mergeAccounts.
	181
2548a7	182	By default, federated repositories can not be pushed to, they are read-only by the isFrozen flag. This flag is ONLY enforced by Gitblit's JGit servlet. If you push to a federated repository after resetting the isFrozen flag or via some other Git access technique then you may break Gitblit's ability to continue pulling from the origin repository. If you are only pushing to a local branch then you might be safe.
831469	183
2c32fd	184	## Federation Pull Registration Keys
JM	185
8c5d72	186	<table>
2c32fd	187	<tr><th>federation.N.url</th>
JM	188	<td>string</td>
3b5289	189	<td>the url of the origin Gitblit instance <em>(required)</em></td>
2c32fd	190	</tr>
JM	191
	192	<tr><th>federation.N.token</th>
	193	<td>string</td>
3b5289	194	<td>the token provided by the origin Gitblit instance <em>(required)</em></td>
2c32fd	195	</tr>
JM	196
	197	<tr><th>federation.N.frequency</th>
8f73a7	198	<td>x [mins/hours/days]</td>
2c32fd	199	<td>the period to wait between pull executions</td>
JM	200	</tr>
	201
	202	<tr><th>federation.N.folder</th>
	203	<td>string</td>
3b5289	204	<td>the destination folder, relative to <em>git.repositoriesFolder</em>, for these repositories.<br/>default is <em>git.repositoriesFolder</em></td>
d7fb20	205	</tr>
JM	206
	207	<tr><th>federation.N.bare</th>
	208	<td>boolean</td>
3b5289	209	<td>if <b>true</b> <em>(default)</em>, each repository is cloned as a bare repository (i.e. no working folder).</td>
2548a7	210	</tr>
JM	211
	212	<tr><th>federation.N.mirror</th>
	213	<td>boolean</td>
3b5289	214	<td>if <b>true</b> <em>(default)</em>, each repository HEAD is reset to <em>origin/master</em> after each pull. The repository is flagged <em>isFrozen</em> after the initial clone.<br/><br/>If <b>false</b>, each repository HEAD will point to the FETCH_HEAD of the initial clone from the origin until pushed to or otherwise manipulated.</td>
2c32fd	215	</tr>
JM	216
	217	<tr><th>federation.N.mergeAccounts</th>
	218	<td>boolean</td>
93f472	219	<td>if <b>true</b>, merge the retrieved accounts into the <code>users.conf</code> of <b>this</b> Gitblit instance.<br/><em>default is false</em></td>
2c32fd	220	</tr>
JM	221
	222	<tr><th>federation.N.sendStatus</th>
	223	<td>boolean</td>
3b5289	224	<td>if <b>true</b>, send the status of the federated pull to the origin Gitblit instance.<br/><em>default is false</em></td>
2c32fd	225	</tr>
JM	226
	227	<tr><th>federation.N.include</th>
8f73a7	228	<td>string array<br/>(space-delimited)</td>
3b5289	229	<td>list of included repositories <em>(wildcard and fuzzy matching supported)</em></td>
2c32fd	230	</tr>
JM	231
	232	<tr><th>federation.N.exclude</th>
8f73a7	233	<td>string array<br/>(space-delimited)</td>
3b5289	234	<td>list of excluded repositories <em>(wildcard and fuzzy matching supported)</em></td>
2c32fd	235	</tr>
JM	236
	237	<tr><th>federation.N.notifyOnError</th>
	238	<td>boolean</td>
3b5289	239	<td>if <b>true</b>, send an email to the administrators on an error.<br/><em>default is false</em></td>
2c32fd	240	</tr>
JM	241	</table>
	242
831469	243	## Example Federation Pull Registrations
JM	244
	245	These examples would be entered into the `gitblit.properties` file of the pulling gitblit instance.
	246
	247	#### (Nearly) Perfect Mirror Example
	248
ea936a	249	![block diagram](fed_mirror.png "Gitblit Mirror")
JM	250
d7fb20	251	This assumes that the token is the ALL token from the origin gitblit instance.
831469	252
93f472	253	The repositories, example1_users.conf, and example1_gitblit.properties will be put in git.repositoriesFolder and the origin user accounts will be merged into the local user accounts, including passwords and all roles. The Gitblit instance will also send a status acknowledgment to the origin Gitblit instance at the end of the pull operation. The status report will include the state of each repository pull (EXCLUDED, SKIPPED, NOCHANGE, PULLED, MIRRORED). This way the origin Gitblit instance can monitor the health of its mirrors.
d7fb20	254
JM	255	This example is considered nearly perfect because while the origin Gitblit's server settings are pulled and saved locally, they are not merged with your server settings so its not a true mirror, but its likely the mirror you'd want to configure.
831469	256
JM	257	federation.example1.url = https://go.gitblit.com
	258	federation.example1.token = 6f3b8a24bf970f17289b234284c94f43eb42f0e4
	259	federation.example1.frequency = 120 mins
d7fb20	260	federation.example1.folder =
JM	261	federation.example1.bare = true
2548a7	262	federation.example1.mirror = true
831469	263	federation.example1.mergeAccounts = true
JM	264	federation.example1.sendStatus = true
	265
	266	#### Just Repositories Example
	267
3b5289	268	This assumes that the token is the REPOSITORIES token from the origin gitblit instance.
831469	269	The repositories will be put in git.repositoriesFolder/example2.
JM	270
	271	federation.example2.url = https://tomcat.gitblit.com/gitblit
	272	federation.example2.token = 6f3b8a24bf970f17289b234284c94f43eb42f0e4
	273	federation.example2.frequency = 120 mins
	274	federation.example2.folder = example2
d7fb20	275	federation.example2.bare = true
2548a7	276	federation.example2.mirror = true
831469	277
JM	278	#### All-but-One Repository Example
	279
3b5289	280	This assumes that the token is the REPOSITORIES token from the origin gitblit instance.
831469	281	The repositories will be put in git.repositoriesFolder/example3.
JM	282
	283	federation.example3.url = https://tomcat.gitblit.com/gitblit
	284	federation.example3.token = 6f3b8a24bf970f17289b234284c94f43eb42f0e4
	285	federation.example3.frequency = 120 mins
	286	federation.example3.folder = example3
d7fb20	287	federation.example3.bare = true
2548a7	288	federation.example3.mirror = true
831469	289	federation.example3.exclude = somerepo.git
JM	290
	291	#### Just One Repository Example
	292
3b5289	293	This assumes that the token is the REPOSITORIES token from the origin gitblit instance.
831469	294	The repositories will be put in git.repositoriesFolder/example4.
JM	295
	296	federation.example4.url = https://tomcat.gitblit.com/gitblit
	297	federation.example4.token = 6f3b8a24bf970f17289b234284c94f43eb42f0e4
	298	federation.example4.frequency = 120 mins
	299	federation.example4.folder = example4
d7fb20	300	federation.example4.bare = true
2548a7	301	federation.example4.mirror = true
831469	302	federation.example4.exclude = *
f6740d	303	federation.example4.include = somerepo.git
JM	304
	305	## Federation Client
	306
	307	Instead of setting up a full-blown pulling Gitblit instance, you can also use the [federation client](http://code.google.com/p/gitblit/downloads/detail?name=%FEDCLIENT%) command-line utility. This is a packaged subset of the federation feature in a smaller, simpler command-line only tool.
	308
	309	The federation client relies on many of the same dependencies as Gitblit and will download them on first execution.
	310
	311	### federation.properties
	312	You may use the `federation.properties` file to configure one or more Gitblit instances that you want to pull from. This file is a subset of the standard `gitblit.properties` file.
	313
	314	By default this tool does not daemonize itself; it executes and then quits. This allows you to use the native scheduling feature of your OS. Of course, if you'd rather use Gitblit's scheduler you may use that by specifying the `--daemon` parameter.
	315
	316	### Command-Line Parameters
	317	Instead of using `federation.properties` you may directly specify a Gitblit instance to pull from with command-line parameters.
	318
	319	java -jar fedclient.jar --url https://go.gitblit.com --mirror --bare --token 123456789
	320	--repositoriesFolder c:/mymirror
	321
	322	java -jar fedclient.jar --url https://go.gitblit.com --mirror --bare --token 123456789
	323	--repositoriesFolder c:/mymirror --daemon --frequency "24 hours"
	324