20,000. Served.
It’s 4am and I’ve just returned home from the now-traditional “Night In The Datacentre” held each semester to ensure the [...]
Posted on December 17th, 2010 by marc
Macquarie University
CIO 2.0: Marc Bailey's blog
"I don't build in order to have clients. I have clients in order to build."
- Howard Roark from 'The Fountainhead' by Ayn Rand, 1942
It’s 4am and I’ve just returned home from the now-traditional “Night In The Datacentre” held each semester to ensure the [...]
Posted on December 17th, 2010 by marc
It’s 4am and I’ve just returned home from the now-traditional “Night In The Datacentre” held each semester to ensure the successful release of student exam results.

This particular night, every single technical element performed in perfect harmony as planned, resulting in serving 20,000 students their results in the critical peak first hour – obliterating the previous record by a factor of roughly 3X.
Although we did have some additional screaming new hardware in the mix thanks to the (well executed) Spike1 project, in fact the mainstream improvement was due to careful planning and incremental lessons learned from the semester one release. Optimising a number of aspects of network, app and web configuration finally allowed us to experience the latent potential of a system design that originated in 2009 and a hardware upgrade executed in 2010.
Student Business and Systems Solutions Director Suzanne Kelly, customer advocate, was there with us in the trenches all night once again and had this to say:
“Wow! What an impressive giant leap…At a time when we’re all working on improving student experience, there was a very significant service improvement tonight in the release of exam results. An Informatics team worked from 11pm-1.30am last semester to analyse where there were areas for improvement, and tonight the team put those improvements in place with an excellent outcome.”
Although we were still ‘slashdotted’ at moments of peak demand, for the first time we can say that every student that wanted access to their result at the moment it was released got it without being gated by University capacity.
We still have more improvements we can make on peak concurrent connections to further reduce the slashdot effect; we were (as it turned out) surprisingly over-conservative this evening by taking time to analyse before dialling up capacity as demand came on stream.
This was an Operations team magic evening, enabled by planning and execution across Programs, Implementation and Products, proving once more that cooperation and collaboration are key to our success.

Particular recognition and thanks to the datacentre team on the night go to Jeffrey Yang, Denis Dudhia, Cy Harrild, Frank Zhou, Bruc Liong, David Moore, Joe Rizzo, and of course Suzanne Kelly.
Disclaimer
This blog is about promoting discussion in areas of significance for Universities it is not necesarily the viewpoint of MQ and should not be taken as such.
© Copyright Macquarie University | Privacy Statement | Accessibility Information
Site Publisher: Macquarie University, Sydney Australia
ABN 90 952 801 237 | CRICOS Provider No 00002J
Comments
Hi All,
It appears to be an awesome result and brilliant improvement.
Congratulations to all concerned.
Cheers
Cathy
Cathy Nelson-Smith - December 17, 2010
You guys have done an awesome job so proud
Dee
Deidre Anderson - December 17, 2010
An excellent result delivering results! Well done. Good outcome for the Uni and good for students. Not involved in this success myself, but I have an understanding of all the bits and pieces that have to be "got right" to deliver this outcome - edge/perimeter servers, MU website, redirection, web application server farm, load balancers, firewalls, network, database, security, etc; not to mention all the business/academic effort to get results collected, verified approved - good to see continuous improvement in action.
Greg Desmond - December 18, 2010
Just out of curiosity - why didn't you use Google Apps to publish the results? After all that's the point of cloud services to provide one off high demand services?
Or are there privacy implications of Google having the exam results?
Thateus - January 4, 2011
Interesting idea; would love to be using a public grid to do this. Unfortunately the world of student admin systems hasn't quite figured out opensource, hadoop, bigtable, google app engine - or scarily - even java. They are large, scary, legacy apps that generally suck and won't make the transition to cloud in their current form.
marc - February 18, 2011