ADMIRAL Sprint 17

We have recently completed our review of Sprint 17.

• Sprint 17 planning: http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_PlanMeeting_17
• Sprint 17 review: http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_17

This review was somewhat overdue, as we've been very busy with follow-ups to ADMIRAL deployment with two additional research groups (making a total of three deployments now).  Main acheivements over the past month have been:

• Two new ADMIRAL deployments; Silk group storage upgraded to use departmental iSCSI facility
• Resolved awkward technical Apache+LDAP issue
• Started construction of stand-alone demonstration environment
• Deployment and management improvements
• Bug-fixing and usability improvements
• Documentation of technical problem areas
• Benefits case study write-up
• ADMIRAL packaging adopted for 1st protoype of Wf4Ever project

One of the recent lessons is that the general level of requirement for data storage has increased dramatically since our initial user surveys.  Where most groups were originally content with 200-400Gb storage, they are now asking for Terabytes (due to increased use of high definition video for observations).  So the ability to connect to a departmental iSCSI network storage facility has turned out to be a crucial development for us, especially for new research proposals that are required to include data management and sharing plans.

Resolving the Apache+LDAP problems has been a most satisfying advance for us; the awkwardness of the Apache web server configuration had been a long-standing difficulty for us, and we will now be able to simplify the overall ADMIRAL configuration and monitoring.

Looking forward, as we enter the final stages of this project, we intend to change our approach to sprint planning.  Instead of preparing a separate plan, we intend to be more reactive, responding to issues in the project issue list (http://code.google.com/p/admiral-jiscmrd/issues/list), as these most closely reflect user feedback and other issues that need to be addressed.  We will still undertake periodic reviews to help us ensure that efforts are sensibly focused.  In addition to dealing with the issue list, two other developments are planned:

• Web interface for user management
• Investigation of Debian installation package for ADMIRAL deployment

The rationale for choosing these is that they appear to be key features for facilitating continued management and new deployment of ADMIRAL systems within the department.

ADMIRAL Sprint 16

We have just completed our review of Sprint 16:

• Sprint 16 planning: http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_PlanMeeting_16
• Sprint 16 review: http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_16

This was the first sprint in Phase 2 of the project.  For phase 2, our focus will be to consolidate and extend the ADMIRAL deployments with research groups, and to ensure that we can continue to support them and thereby gain greater understanding of local-level data management concerns through other projects to be conducted over the coming years.

Much of our effort in this sprint has ben focused on deployed system quality improvements that allow us to confidently roll out further ADMIRAL deployments, and in particular to separate configuration data from from the deployed system software so we can update the software without too much disruption to the ADMIRAL users.

Specific enhancements completed in this sprint include: 

• Dataset repository submission tool usability enhancements
• Improved deployment scripts to facilitate software updates
• Improved test suite and generally improved system robustness
• Tested ADMIRAL with a new departmental storage server

With these quality improvements mostly completed, our next immediate goal will be to extend the use of ADMIRAL by our research users.

ADMIRAL Sprint 14 review

A review of sprint 14 can be see at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_14.

Progress has been quite satisfying, with several scheduled activities completed, and more progress than planned on the ADMIRAL-to-Databank dataset submission tool,  all in spite of experiencing a disk failure on one of our servers that took about 1.5 days effort to recover. This is balanced by less-than-planned progress on deploying ADMIRAL for the Development Group, and not yet having researcher feedback on the dataset submission interface.

We are still awaiting a finalized Databank API test suite.

A new ADMIRAL instance for the Evolutionary Development has been built and deployed, but has not yet been finally configured and handed over for use by them.

We have a functional dataset submission tool and web interface, but there are some significant usability problems that need to be adddressed before we want to consider showing it to researchers. This remaining work is mainly in the area of dataset selection, and the required server components have been developed and tested: it just remains to update the web page to use the new capabilities.

As intended, we completed work to revise the ADMIRAL local store test suite, before scheduling the sprint reviewed here.

ADMIRAL Sprint 13 review

A review of sprint 13 can be see at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_13.

In summary, Progress has been quite satisfying, with most scheduled activities completed, and also some unscheduled progress. Our new recruit is coming up to speed well with the project development and testing activities.

The initial dataset selection and display functions that will allow researchers to view their submitted datasets are functionally complete (apart from finalizing their deployment via the RDF Databank).

We have made substantial progress on updating the local file store system build scripts and test suite, with a view to building a system for use by the Evolutionary Development group.

Some progress has been made on updating and finalizing the Library Services RDF Databank service, but we are still awaiting a version that we can test and use as a target for dataset submission scripts.

For ongoing work, our immediate priorities are to (a) finish updating the local file store test suite, and (b) configure and deploy a system for the Development group. Beyond that, we need to focus on finalizing the RDF Databank API test suite, and building tools to allow researchers to select and submit datasets to the Databank.

ADMIRAL Sprint 11 review

This sprint has been substantially taken up with introducing a new developer to the various ADMIRAL technologies.  Progress (not enough) has been made with improving and debugging the ADMIRAL system generation scripts, but the resulting system is not passing its tests.  Some progress has been made to creating a utility for displaying dataset content in user-friendly fashion.

Review of sprint 11: http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_11

ADMIRAL Sprint 10 review and Sprint 11 plan

A new (half-time) developer started on the ADMIRAL project yesterday. After the usual administrative details, and setting up as development environment, we did a mini sprint plan for the next 2 weeks of the ADMIRAL project.  I say a mini-sprint plan, as we didn't do the full activity/user story selection, task breakdown and scope bartering, but rather reviewed the remnants of the most recent active sprint plan and identified key unfinished tasks to be tackled.

The next goal for the project is to complete the functionality covered by phase 1 of the project plan by the end of October, with front-to-back submission of research datasets to the Library Services Databank repository service, and providing visible web-based feedback to our research partners of the submitted datasets.  This we intend to use as the basis for iterative improvements and enhancements in phase 2 of the project, with the researchers guiding us concerning what constitutes useful metadata to capture and expose with the submitted datasets.

The sprint plan for the period to 7 September aims to:

• review, debug and update documentation for the ADMIRAL system scripted creation procedure
• create a new ADMIRAL file sharing deployment for the evolutionary development group
• file store bug fixes (password over unencrypted HTTP channel; https access reporting server configuration error
• progress work on Shuffl RDF support

Review of sprint 10:
http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_10

Plan for sprint 11:
http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintPlan_11

We've also had a technical meeting with the Library Services developer of RDFDatabank (aka Databank), the data repository system:
http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_20100825_technical_meeting_with_library_services

ADMIRAL Sprint 8 Review

The review for sprint 8 is posted at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_8.

Project activity has been impacted by unplanned staff absences, but useful progress has been made on a number of technical fronts. Interaction with Library Services to develop the data repository service has been going well, and we look forward to rapid progress in this area. Work to add RDF serialization to Shuffl is under way. Rolling out ADMIRAL data stores to additional research groups has been stalled.

This review is being posted nearly a week later than planned, so includes a period not covered by the original sprint plan.

Summary of achievements this sprint:

• project partner and OULS meetings
• attendance at JISC TransferSummit meeting
• brief project description for D-Lib magazine
• Shuffl WebDAV file browsing implemention complete
• RDFDatabank access
• RDFDatabank initial test suite
• negotiating revisions to RDFDatabank API in light of experience
• started on RDF serialization for Shuffl

ADMIRAL Sprint 7 Review

The review for sprint 7 is posted at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_7.

Progress has been achieved on a number of fronts, and we are now ready for access to a test deployment of the OULS Databank repository system. The ADMIRAL data store is in use by the Silk research group.

We attended the JISC managibng research data programme meeting, and have established lines of potential close collaboration with the Oxford-based SUDAMIH project, which were further developed at our project advisory board meeting [1].  We also discussed collaboration with CLARION, particularly in the area of integrating their embargo manager into a different environment.

Summary of achievements this sprint:

• Project and steering group meetings;
• attended JISCMRD programme meeting;
• Shuffl access and browsing of to ADMIRAL shared files;
• abandoned attempt to use Ubuntu 10.04;
• sample code for creating BagIt (Zip) packages;
• enhancements to user management scripts;
• initial investigation of RDF parsing using jQuery.

[1] http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_20100520_advisory_board_meeting

Sprint 6 and progress-to-date review

Sprint 6 has been generally fruitful:

• Silk Group file sharing is ready for use;
• Silk Group file sharing online user documentation has been prepared;
• Shuffl WebDAV storage module has been implemented;
• Shuffl is now able to run from and save workspace data to an ADMIRAL system;
• HTTP authentication works well with AJAX calls;
• JISC 6-month report has been submitted.

The full review is available at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_6.

This review follows a different pattern to the usual end-of-sprint review.  Having a data sharing platform ready for use by researchers was originally estimated to take about 1.5 months, but the work has actually taken about 3 months. The question driving this review is: "Why so long?".

In summary, we don't feel the execution has been greatly lacking (though other experts may disagree and we welcome discussion).  The mismatch between plan and execution seems to be due to oversights in the planning. Some lessons we offer are:

• We should not assume that using tried-and-tested software means that there is no development risk.
• We need to create a system that provides the user with a smooth, secure interface. Focusing on the user experience can easily push software components out of their design "comfort zone".
• Plan for an extended period of "scoping experiments" with any external technology, to be conducted alongside user requirements gathering.
• Plan to script as much as possible of a system build and configuration: this takes longer to begin with, but saves time in the long term as it facilitates repeating a build with minor variations.
• Plan to create automated tests for as much as possible. It may seem stange to be creating "unit tests" for accessing a file system, but they have been invaluable for our work in testing access control mechanisms.
• Plan to build separate systems for testing and live deployment. Automated tests should run with either.
• When working with a number of users, plan a separate system build and deployment phase for each.
• We estimate that ongoing user interactions require about half a day per week.
• Estimated project management overheads (for a small project on the scale of ADMIRAL): 25% for the project manager, and 10% for other team members.
• We estimate that about 20% of our development time is spent on creating technical documentation through wikis and blogs, but this is generally absorbed in the development effort estimates.

Projects aimed at changing user behaviour, such as ADMIRAL, are critically dependent on a solid technology foundation. Despite not being focused on technology development, we cannot dismiss the engineering effort that is needed to create a smooth, dependable experience for the users.

We estimate that access to in-depth expertise about the software we were using may have saved 25%-50% in some of the areas of time overrun.

Sprint 5 review

We've just reviewed a particularly long sprint.  Notes of the review are at http://imageweb.zoo.ox.ac.uk/wiki/index.php/ADMIRAL_SprintReview_5.

(We chose to run a long sprint with an intermediate review because we feel that having sprints less than 2 weeks creates too much planning overhead in our particular circumstances - reviewing and adjusting the plan half way through the sprint seems to be a workable compromise.)

The primary goal of the sprint was to complete deployment of a live ADMIRAL file sharing service for the Silk Group, based on a new virtual hosting environment.  We fell just short of that goal, mainly due to having to introduce new, unanticipated, access control mechanisms to handle the group's specific requirements.  But in the process of doing this, we seem to have developed a solution to a problem for which the conventional wisdom seems to be "you can't do that", viz. effective file access control for files that can be created via HTTP/WebDAV ''or'' locally or by CIFS. The new ingredient that makes this possible is Linux ACLs.

An initial draft security model for ADMIRAL has been created, though will surely benefit from review and revision as the project proceeds.

Maybe the biggest disappointment of this sprint was our failure to convene an intended monthly meeting, due primarily to unavailability of key partners, and possibly also because of not allowing enough lead time for planning.