Hey Francois,

My initial findings with using the plug-in:

Setup was pretty straightforward but I did have one issue on our Windows 2003 server, around authentication with AccuRev: it complained that the AccuRev session hadn't been authenticated yet despite the fact I had provided the appropriate username/password in the VCS root settings. I managed to get it to work by logging onto the server console and manually doing a "accurev login username password".

After that I had no trouble setting up a project and build configuration using the new AccuRev VCS root. After the initial population of the work area, it was extremely fast in retrieving changes when I promoted to the stream specified in the VCS root settings. Check-out rules and trigger patterns worked a treat which was very encouraging.

However when promoting to any parent streams of the specified stream it didn't pick up the changes. I presume this is by design, as usage of the HIST command against a stream will only ever return changes directly on that specific stream. Is that the intended behaviour? If so, had you any thoughts about practises within TeamCity to support the AccuRev stream inheritance model?

Wish list for the plug-in:

• Ability to specify an optional "watch" Stream in addition to the Stream setting which the plug-in uses as a basis of the working area it populates. For example, say I wanted to watch for changes to Stream A, but I want TeamCity to actually populate the work area from a child stream; Stream B. In this scenario, Stream A would be the "watch" Stream.

• Throw error on overlap detection. I'm not sure how possible this is but it would be great if TeamCity could show an error in the project list if one of the VCS roots has an overlap (i.e. conflicting changes) that will require a merge in AccuRev. The idea being that overlaps can very often cause issues in builds.

• Poll based on AccuRev trigger. This may not be possible due to limitations in TeamCity, however it would be nice for the plug-in to only look for changes when told to do so by AccuRev, as opposed to polling the server at regular intervals for each VCS root (not sure if there are any issues yet with continuous polling in AccuRev).

I'm sure I'll think up some more later but again I'm really encouraged with the alpha and look forward to seeing this develop!

Francois, thanks for this information.  I wanted to give you the heads-up that AccuRev 4.7.1 will have enhancements to the command-line which will help you detect changes in a stream that have come through inheritance.  This is really geared toward continuous integration applications and will greatly assist in plug-ins like yours.

Keep an eye out for the 4.7.1 release in the next couple of weeks.

Cheers,

~James Talbott

Senior Systems Engineer

AccuRev, Inc.

Hi James,

Did this feature make it into 4.7.1? I can't see it mentioned in the release notes.

Thanks,

Jamie.

Daniel,

Actually, we already have the feature filed for VCS changes pushing. Please also see the issue comments for a way to configure the approach with the current TeamCity version.

Also, feel free to vote for the feature.

--
Best regards,

Yegor Yarko
Project Manager (TeamCity)
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

I might have a scenario that was never thought through.  I work in an environment which has multiple AccuRev servers (different machines).  The only way I can see a single TeamCity installation working with both servers is if every call included the server name parameters ("-H server:port").  In the AccuRev code for RunProcess, there exists a setter method called setSessionToken().  If this parameter is set, the server name gets appended to every AccuRev call in the method addSecurityArgsToCommand().  Without it being set, no server information is appended.  In the TeamCity AccuRev plugin, I see no reference to this method call.  Is there a reason why I can't find it?  Can someone look into this and add it?  I may end up playing around with adding it myself locally.

Thanks

Hi Francois,

I rarely read documentation for the reason that it is oh so often incomplete.  I go straight to decompiling.  I am using the exact same version of the AccuBridge library that you've included in the plugin.  I'll email you directly my code.  I worked on this yesterday and got it working.  Look for files with a July 13th timestamp and do a diff between them and your version to see what I've changed.  If you merge the changes into the trunk and create a new build, let me know so that I can convert back to your build.

thanks

Francois,

I've had a try at incorporating the AccuRev DIFF command into the plugin.

One challenge I encountered is that the command actually returns a flat list of changes for a given Transaction range. What we really want to know is on what stream(s) the changes were made, when, by whom, etc.

The solution I implemented is as follows:

1. When the "CollectBuildChanges" method is called, it performs the DIFF command using the "from Transaction" to the most recent Transaction ID within the entire depot.
   
2. If any Changes are returned, it performs the HIST command on the specified stream (as per the current version) using the same Transaction ID range, and attempts to match each DIFF Change against a Transaction that is returned. Any found Changes are flagged as "matched" and added to the Modification collection.
3. If there are any Changes left that are not flagged, then it recurses up a level in the stream hierarchy and repeats Step 2. The recursion continues until either the root stream is encountered or all Changes in the collection have been flagged.
4. If there are still Changes left after encountering the root node, it adds the remaining Change items to a single Modification item with the description "Other Inherited Changes". This will tend to happen if you make a change to a stream, e.g. changing the stream's basis time.

We've been trialling/testing the changes in our shop for the past week and so far so good. In particular the devs really appreciate being able to see the inherited parent stream changes flow down to their builds. One nice side effect of this is that having a "watch stream" is now possible, due to the fact the plugin will now collect changes all the way up the stream hierarchy. Also, using AccuRev include/exclude rules on the watch stream has become a very nice alternative to TeamCity VCS checkout rules because it will directly affect what gets populated in the TeamCity work areas and also in the AccuRev DIFF lists.

One last thing I need to do is to get my site working with Derrick's "set token" changes (for some reason it won't authenticate for me at the moment). Please let me know if you'd like to try the changes and possibly put them into the trunk.

Cheers

Dan

I just tried upgrading our TeamCity server to use the newest version of the AccuRev plugin (we were previously using 0.2), and this version breaks for me.  When it tries to run the "accurev update" command, the command fails and prints its help page as though it doesn't recognize the arguments that are being passed to it; I think it doesn't recognize the "-p" option (at least, I don't see that option in any of the official documentation, and it fails when I try to run it manually).  This leads to the AcRunCommand class getting an IndexOutOfBounds exception when it tries to parse the XML output.  I made a bug case here: http://youtrack.jetbrains.net/issue/TW-9872

Any idea what might be going on?

Confirmed the issue was in the 'clean build' option; was a victim of some last minute refactoring of mine.

Now fixed in latest build: http://teamcity.jetbrains.com/viewLog.html?buildId=26515&buildTypeId=bt138&tab=artifacts

A bug which can cause failure of change list population and prevention of build execution has just been fixed.

We discovered the issue this week when element ID of a directory/file changed during a 'move' operation.

Build available here: http://teamcity.jetbrains.com/viewLog.html?buildId=29123&buildTypeId=lastSuccessful&tab=artifacts

Hi Phillip,

Apologies for that, unfortunately I don't have a way of subscribing to those YouTrack bugs at the moment.

Regarding the issue, I'm not entirely sure what's going on here. Can I suggest you try manually running this command on your TeamCity build server and let me know if the permissions are retained?

accurev pop -H [accuRevServerName] -fx -R -O -v [streamNameToPopulateFrom] -L [directoryToPopulateInto] \.\

e.g. accurev pop -H accurev:5050 -fx -R -O -v FR -L E:\Program Files\TeamCity\temp\accurev6205070683 \.\

This is the same command that the plugin uses to initially populate the vcs root's working area.

By the way I've noticed that TeamCity seems to populate a separate directory to where the build actually takes place, i.e. it first populates a TEMP directory similar to the one mentioned here and then transfers those files over to WORKING. Not sure if anything is being lost in that file transfer but it's something to bear in mind.

Dan

That's interesting. Just to be absolutely sure about this though, do you use the "clean all files before build" option on the Version Control Settings tab (this causes the ACCUREV POP command to be called every time, instead of an UPDATE)? If not if you could try that and confirm the permissions are still being lost... that would be a good case for JetBrains to take this up.

That's cool, I was mainly interested just to try it to see if the problem was limited to the UPDATE command.

So if its not the update that is responsible, the culprit is more likely to be the plumbing between the agents and the central build server. Unfortunately this is all controlled by TeamCity so I'm not sure how much can be done with the plug-in. Perhaps this is worthy of a broader post on the TC forums?

Daniel,

Actually, there seems to be a way to subscribe to issue events in YouTrack.

You can try the following:

• Search for the issues you are interested in (e.g. #{3rd party: AccuRev plugin} #Unresolved)
• Click "Save this search" below the search field and give a name to the search.
• Go to your Profile, Filters and Notifications and tick necessary options for the Saved Search.

Has anyone tested the latest plugin with TeamCity 5? We recently setup a TeamCity 5 server, running under Linux, I have installed the latest AccuRev plugin, but it doesn't seem to detect changes anymore. When I click build it does seem to pick up the correct source code, but the monitoring of changes doesn't appear to be working.

Previous I was using TeamCity 4.5 on a Windows box, so there are a number of things that have changed now - not sure which one has caused this issue?

Any ideas?

Steve

Fantastic.

Thank you.

Hopefully your endeavours with your test Windows box will issolate if the issue is with TeamCity 5 or with the fact I am now running a Linux based server now.

Steve

For what it's worth, we're running TeamCity 4.5 on an Ubuntu Linux 8.04 server with the AccuRev plugin, and we haven't encountered the issue mentioned above.  In fact, the issue we have is practically the opposite -- sometimes it will detect changes in the source stream but not actually download the new source code. :-)  The case is detailed here: http://youtrack.jetbrains.net/issue/TW-10825?query=%23{3rd+party%3A+AccuRev+plugin}+%23Open

I'm glad to hear that Steve, thanks for the update!

Regarding the checkout problem, I have finally got off my lazy behind and raised this with AccuRev.

One thing that may really help is if anyone experiencing the problem could post snippets of their "stdout_yyyymmdd.log" file (available in the "<TeamCity Data Dir>\Logs") from around the time that the bug occurs. It might be a bit of to-and-fro to get this one resolved.

For anyone interested, the workaround for this we use is to create a sub-stream under the stream we want to checkout from. We then add include/exclude rules to that sub-stream so that only the elements that are required for that particular build are included in the checkout. We then setup the TeamCity VCS root to monitor that sub-stream, which in turn watches all of its parent streams. This generally results in much faster build times because only the required files are monitored for changes and retrieved during the "clean build", rather than the entire depot. This has helped us keep all of our build processing times < 4 minutes total. I can understand that this may not work for everyone, though.

I have augmented to AccurevPlugin to support agent side checkout.

This has reduced the amount of time required to perform a checkout from 16 Minutes to under 2 minutes.

We would like to get theses changes integrated into the official plugin and were wondering how to proceed.

Please advise:

Thank you

For what it's worth, I would be eager to help test this if any help is needed. :-)

A quick question about the implementation of agent-side checkout: does it account for AccuRev possibly being installed in different locations on different build machines?  On some of ours it's in /usr/local, on others it's in /opt... it'd be great if it looks for it either in the PATH or if there's an agent-specific property that can be set to indicate where it is.

We are just putting the finishing touches to the plugin.

I will send you the source code once we have eliminated all the major bugs.

Thanks

Thanks!  I upgraded the plugin on our server and so far everything seems to be working well.

Based on my observations, it seems like when build configurations are set to checkout-on-agent mode, every time a build runs it re-populates the directory from TeamCity's build stream.  This can take a while for large streams, and waiting a few minutes for it to repopulate when only a few files have changed seems a bit excessive.

It seems to me like it would be a good idea to, in addition to creating a stream for each build agent and each VCS root, also create a reference tree off of that stream so that the "accurev update" command can be used to quickly change between revisions without repopulating the directory.

Not all of our build agents are on the same network segment as our AccuRev server (in fact I don't know if any are...), and some of them have to go through 100 Mb/s links, so it's not necessarily very fast -- we've got one 1.1 GB source tree that I regularly see 3-4 minutes to populate, and a 3.6 GB tree that takes 10-12 minutes.  There are other trees, too, but those are getting the most activity at the moment.  That same 3.6 GB tree has 61 build targets that depend on it, so it takes a long time to do a complete build if it repopulates the work directory for every build. :-)

Maybe I'm not clear on something -- is the AR plugin cleaning out the source tree between every build?  I wouldn't expect that to happen unless the "Clean all files before build" flag is explicitly turned on.  In fact, we have a few build scripts that expect the artifacts from the previous build to still be there.  (I know, it's bad, but they're legacy scripts that were in place long before we started using TC, and we haven't had the time to rewrite them.)  If the agent-side checkout code was using a reference tree for building, I would expect it to just do an "update" between builds and leave any modified or external files in place unless it was specifically told to clean the agent's sources.

In practice, if we ignore any modified or external elements and simply use "accurev update" to move between different versions of a workspace or reference tree, how long it takes only depends on how much has changed; it only takes maybe 10-15 seconds to update a workspace that's had only a few changes.  In the worst-case scenario, it takes about as long as a normal full populate would.  We have our build configurations in TC set up to automatically rebuild whenever something in the appropriate directories has changed, so usually changes to the stream between builds are minimal; when using the checkout-on-server mode, it usually takes maybe 10-15 seconds for the server to acquire the changes and copy them over to the build agent.  I haven't done any specific testing with reference trees, but I would expect them to be about the same; from what I understand, they're basically like read-only workspaces that are specifically intended for automated building.

If you actually need to do a complete wipe and repopulate of a workspace with the latest code from its parent stream, probably the easiest way to do that is to just delete the whole directory, then run "accurev update -9" and then do a populate.  If you're not familiar with "update -9", it's an undocumented flag that marks a workspace as being "updated" in AR's database but does not actually change any files on the disk.  Its use is usually not recommend, but sometimes it's helpful.

We have a reasonable sized code base in AccuRev, so we have filter streams to get rid of a lot of unnecessary content for a given project, and then have everyone parent off of the filter stream. The problem we found previously was when attached to the filter stream we didn't see changes from above, but as you say, this has now been fixed, but something else we (well Abe) is adding for support for now is the ability to promote on success (optional), which wouldn't work if we just pointed directly at the filter stream (there'd be nothing to promote in the filter stream). So we end up having it something like:

Watch : ProjectA_GAMEBUILD

Populate : ProjectA_GAMEBUILD_Filter

And if that succeeds, it automatically promotes the code in ProjectA_GAMEBUILD, up to the transaction tested.

Maybe there is a better way to do all this? It is what we have been doing up till now (with out own batch files to handle the AccuRev side), so we are just trying to get it a bit more integrated with the plugin.

On a side note, we have noticed the 'seeing changes from above streams' doesn't stop at snapshots (not sure about time based streams) - we were going to have a look ourselves at fixing this, unless it is something you are already fixing, or know can't be done due to AccuRev limitations etc.?

Steve

Thanks Abraham,

I'll take a look at the changes you've provided.

By the way, I've meant to mention that I've been in contact with AccuRev regarding the incremental workspace update problem (i.e. the issue that occurs when the "clean all files before build" option is not checked).

They have indicated that the method that was implemented in the plugin is not supported. The method that is used is basically a single "ACCUREV UPDATE" CLI command with a transactional range (From Transaction A to Transaction B/C/D/E/etc), however AccuRev have indicated that only one transaction can be specified at a time using this command.

This would explain why on some occasions changes don't get retrieved from the repository, because in order to do so you would need to make individual calls to "ACCUREV UPDATE"  to retrieve changes in Transactions B, C, D and E.

I'll have a look at implementing this change when I have some downtime, unless of course someone else is keen to have a go!

10 GB, wow that's quite a lot. What is it you work on, if you don't mind me asking - I'm just curious!

The agent side plugin was just developed to meet our needs initially, which is to do a fresh AccuRev populate of the given stream at the given transaction. We have about 1 GB of source code and it takes a minute or two to populate it, I don't think you'd want to do that every time with 10 GB of data. So I think the first thing you'd want on the agent side checkout side of things is to implement an incremental checkout option. We were going to look into this by using reference trees, but as of yet we haven't started on this task (and hadn't got it penciled in for anytime too soon). Ideally this would also make sure the source tree ends up correct, e.g. if the build processes modified a sources file, that source file should be reverted etc.

There are probably a few other enhancements to the plugin that could be made, if I think of any good ones I'll post here!

Steve

Hi all,

I'm very happy to say that we've just started work on improving the Accurev plugin to cover our use cases.

Initially that is to add a couple of things:

Agent-side checkout

This is a must-have for us; right now we are doing this using a python script on the agent, which works but isn't neat and throws off build timing (as the update/sync time is counted as part of the build time)

We handle building from a specific Accurev Transaction by creating a timelocked stream and workspace, as this is currently the only way to get this behaviour in Accurev.

Limited support for checkout rules

We want to specify that Accurev syncs to a sub-folder of the TeamCity Checkout directory, because for some builds we have checkouts from both Accurev and Perforce and obviously don't want them both writing to the same directory.

Right now the Accurev plugin doesn't seem to support the rule we want, so we'll be adding support for it.

In the future we might add more checkout rule support e.g. for including/excluding things from a Workspace based on the checkout rules, but for now we'll be implementing that by using additional Streams in Accurev and applying the include/exclude rules to them.

I'm hoping to have both of these working in the next couple of weeks, so should be able to update the plugin with our changes soon after that.

Cheers,

Mark

We are currently using the trunk version, which I had thought included the zip files posted here but now I realise they don't. Oops!  Is that in branch somewhere already? We're going to take a look at the version from Abraham and most likely start from there.

We've already noticed the plugin is using the "old style" VCS APIs, but figured we'd get it working with those first and then look at porting over to the new API as a separate task.

Hi Folks,

I have tried to enable the VCS labeling option for Accurev and it seems to have no effect. I have selected the "Successful only" radio button and checked the VCS root to label.

Are snapshots supported in this plugin?

Best Regards,

Daniel

Hi Dan,

You're right... the stdout _YYYYMMDD.log file shows all accurev commands issued during a build.

An mksnap command is being executed which works perfectly well from the command line, but doesn't seem to make any difference during a TeamCity build... strange.

I'll take a closer look at this on Wed (tomorrow is a national holiday in Brazil) and will post here any findings I happen to come across.

Cheers,

Daniel

Hi Abraham,

I have a number of changes to commit myself.

We've not cleaned them up fully yet but I can quickly bundle up what we've got an post here as a zip.

UPDATE: attached source and prebuilt zips

Cheers,

Mark

Excellent stuff, thanks Daniel!

Hi Pete,

There are no current plans to introduce Pre-Tested commts/Remote runs in the plugin.

I am the only developer remaining and unfortunately my time is needed elsewhere at the moment.

If you or anyone else would like to step forward and take the lead on the OSS project please let me know, and I can help get you access to the JetBrains-based repository and builds.

Regards,

Daniel