Just as a quick little public announcement - if you are having problems with your subscription to kde.org mailing lists, you may wish to check that your provider does not use SORBS to filter your email.
It appears that our email server has (for one reason or another) been listed on their blacklists. We are currently working on getting delisted, but that may take a few days unfortunately.
Thursday, November 3, 2011
Thursday, August 11, 2011
Our git hooks and how to report problems
On occasion it has been known for the git hooks to trip up when handling your commits. When they do trip up, they will usually exit out - rejecting the push. Unfortunately, fixing these issues isn't always easy. When the hooks trip up, the actual error which is causing it has usually occurred much earlier in their execution.
While tracing the exact origin of when this error actually happened is usually fairly easy, actually determining how to fix it is much harder. This is especially the case when the bug lies in the information retrieval component, which uses many complex regular expressions to do it's magic.
Please, when reporting bugs with the hooks - provide the commit which is causing the hooks grief. It makes the process of identifying what has broken - and fixing it - much faster.
While tracing the exact origin of when this error actually happened is usually fairly easy, actually determining how to fix it is much harder. This is especially the case when the bug lies in the information retrieval component, which uses many complex regular expressions to do it's magic.
Please, when reporting bugs with the hooks - provide the commit which is causing the hooks grief. It makes the process of identifying what has broken - and fixing it - much faster.
Monday, June 13, 2011
Git - and some of it's more problematic interfaces
(Warning - this post contains some mildly technical content surrounding git)
As some of you may already know - KDE's Git hooks are not capable of detecting copying or moving of files at the moment. They have also been known to show some rather strange and confusing codes concerning file changes too (The "I" code). The origin of these deficiencies lies deep within git itself, and why git is like this is a mystery to me.
When the hooks first start to consider sending out emails, they have a number of bits of information available to them. One of the most vital pieces of information is the "name-status" of files that changed in the commit. This tells the hooks if the file was added, modified and so forth. Unfortunately it misses out one crucial detail - the number of lines added or removed from the file. This information is a must have for the message contained in the emails it sends - so it needs it.
In order to get this information, the hooks call out to git, and recieve a "numstat" in return. This is where the problems begin. Whilst the numstat may look at first sight to be normal, it is only when copies, renames or merges come along that the trouble begins. To see how things can be problematic, let's compare the output - when moving files.
First, from name-status:
Looks normal enough, and my initial parser in the hooks handled this fine. Now, let's look at the numstat:
As you can see, it is completely different. When I first tried to parse this, I came across a number of obstacles, one of which concerned files containing { or } characters. This made the parsing process dangerous and unreliable at best unfortunately.
The other major difference in the numstat is much more confusing however. With a merge, unless you had to intervene by hand to fix a conflict - the name-status will show nothing. Yet the numstat will show a number of changes. When I first tested the hook code, this caused it to bail out, which is where the "I" code comes in. This code is quite simply a hack which works around a weakness in git, as the hooks try to glue all this information together.
This unusual - and very different behaviour of two git commands is not documented anywhere in the man pages, and no option exists to give a behaviour more like the other. The only reason I can guess for this unusual behaviour is a hack in git itself - to give a nice output when you merge or fast-forward. That is just speculation however.
For the future, I can't see the current state of the hooks changing too much unless git itself changes. Until that happens the support for detecting file moves or copies, and giving completely consistent output will not happen unfortunately.
As some of you may already know - KDE's Git hooks are not capable of detecting copying or moving of files at the moment. They have also been known to show some rather strange and confusing codes concerning file changes too (The "I" code). The origin of these deficiencies lies deep within git itself, and why git is like this is a mystery to me.
When the hooks first start to consider sending out emails, they have a number of bits of information available to them. One of the most vital pieces of information is the "name-status" of files that changed in the commit. This tells the hooks if the file was added, modified and so forth. Unfortunately it misses out one crucial detail - the number of lines added or removed from the file. This information is a must have for the message contained in the emails it sends - so it needs it.
In order to get this information, the hooks call out to git, and recieve a "numstat" in return. This is where the problems begin. Whilst the numstat may look at first sight to be normal, it is only when copies, renames or merges come along that the trouble begins. To see how things can be problematic, let's compare the output - when moving files.
First, from name-status:
R100 repo-configs/audit/kaudiocreator.git/skip-author repo-configs/audit/scratch/nalvarez/mobipocket-migration.git/skip-author
R100 repo-configs/audit/kaudiocreator.git/skip-eol repo-configs/audit/scratch/nalvarez/mobipocket-migration.git/skip-eol
R100 repo-configs/audit/kaudiocreator.git/skip-filename repo-configs/audit/scratch/nalvarez/mobipocket-migration.git/skip-filename
Looks normal enough, and my initial parser in the hooks handled this fine. Now, let's look at the numstat:
0 0 repo-configs/audit/{kaudiocreator.git => scratch/nalvarez/mobipocket-migration.git}/skip-author
0 0 repo-configs/audit/{kaudiocreator.git => scratch/nalvarez/mobipocket-migration.git}/skip-eol
0 0 repo-configs/audit/{kaudiocreator.git => scratch/nalvarez/mobipocket-migration.git}/skip-filename
As you can see, it is completely different. When I first tried to parse this, I came across a number of obstacles, one of which concerned files containing { or } characters. This made the parsing process dangerous and unreliable at best unfortunately.
The other major difference in the numstat is much more confusing however. With a merge, unless you had to intervene by hand to fix a conflict - the name-status will show nothing. Yet the numstat will show a number of changes. When I first tested the hook code, this caused it to bail out, which is where the "I" code comes in. This code is quite simply a hack which works around a weakness in git, as the hooks try to glue all this information together.
This unusual - and very different behaviour of two git commands is not documented anywhere in the man pages, and no option exists to give a behaviour more like the other. The only reason I can guess for this unusual behaviour is a hack in git itself - to give a nice output when you merge or fast-forward. That is just speculation however.
For the future, I can't see the current state of the hooks changing too much unless git itself changes. Until that happens the support for detecting file moves or copies, and giving completely consistent output will not happen unfortunately.
Tuesday, January 18, 2011
Changes in KDE Git infrastructure
As some of you may be aware, sysadmin is currently preparing to launch a big change to git.kde.org: New hooks. These new hooks will be launched tomorrow, so the repositories may be in maintenance mode for a period of time.
The biggest improvement to these hooks is commit mails, which have improved significantly. Particular highlights of this is the format of the mails, and the extra information about what changes took place being included.
An example of a commit mail can be found here.
The upshot of this is that commit filters on sub-directories of git repositories now work, and the size of messages is now limited.
Several new headers, X-Commit-Ref, X-Commit-Project and X-Commit-Folders have also been added, which should hopefully allow for a more git native version of commit filter to now be built.
Regarding the format of the mails, this has changed to be more similar to SVN emails, and includes information on who pushed the revision to the repository.
Another important step forward is the launch of KDE Projects xml output. It provides a variety of information on projects, which should hopefully be usable by scripts to build a tree of clones on developer systems. This output is available here and is updated every 5 minutes.
The biggest improvement to these hooks is commit mails, which have improved significantly. Particular highlights of this is the format of the mails, and the extra information about what changes took place being included.
An example of a commit mail can be found here.
The upshot of this is that commit filters on sub-directories of git repositories now work, and the size of messages is now limited.
Several new headers, X-Commit-Ref, X-Commit-Project and X-Commit-Folders have also been added, which should hopefully allow for a more git native version of commit filter to now be built.
Regarding the format of the mails, this has changed to be more similar to SVN emails, and includes information on who pushed the revision to the repository.
Another important step forward is the launch of KDE Projects xml output. It provides a variety of information on projects, which should hopefully be usable by scripts to build a tree of clones on developer systems. This output is available here and is updated every 5 minutes.
Subscribe to:
Posts (Atom)