Before digital archives and cloud storage, the nation’s scientific research data was stored in vulnerable hard copies—one fire and vital documentation of plant and animal life across the country would have been lost. But is our valuable information that much safer now? Without consistent system updates and comprehensive biodiversity data security measures, it may not be.
In 2020, our Federal Government client – a group that manages a system holding Australia’s biodiversity data – sought to mitigate this weakness in our modern documentation process by upgrading their system and migrating the outdated environment onto AWS. Navigating the complex transition without jeopardising the vast amount of data in their care required a trusted partner—Blue Crystal Solutions.
Nationally significant Biodiversity Data Security at risk in aging system
Historically, all of the clients infrastructure was run through on-site servers. After a period of retirements and changing skillsets within the team, moving onto CSIRO infrastructure was chosen, and ran there for several years. However, due to organisational reasons with CSIRO, it became imperative to move the client back onto their own infrastructure.
The problem was, their current software was dangerously out of date, with applications that could stop running at any moment—putting the extensive scientific data they held at risk.
Australia-wide records that demanded safe keeping
In the case of the Australian National Botanic Gardens, the data spans 40 years and is used nation-wide for conservation management. The client also holds records of the preserved material at the Australian National Herbarium and the National Species List.
If the clients system had failed, all of this nationally significant scientific data—documenting the vital details of all plant and animal life in the country—could have been lost or compromised. The client was faced with the challenge of upgrading the software from the ground up and figuring out how to ensure the ongoing sustainability of maintaining the infrastructure.
A cloud platform fit for the project
AWS was the chosen cloud platform from the beginning. When those running the transition examined the issue, AWS was chosen as the new cloud platform as best fit for the kind of infrastructure the client was running and the types of projects they were supporting.
Recommendations within the Government
Blue Crystal was already a trusted partner for other sections of the government, and it was their existing relationship with Higher Education Services (HES) that lead them to the client. CSIRO and HES recommended Blue Crystal’s expertise and they were selected as the supplier for transitioning the client onto an AWS-hosted environment.
The complexities of transitioning from an outdated system
With so much significant data at risk, there was no room for mistakes. So, Blue Crystal began by mapping out the client’s current environment:
- Running workshops to discover use cases
- Completing a POC to prove key risks
- Identifying the ideal architecture design and necessary costing
The client was running on old, out-of-date servers that made the transition complex—it was up to Blue Crystal to determine how to lift-and-shift the system smoothly onto AWS. The team made the pragmatic decision to recreate the environment from their original system in AWS, ensuring they could move all the data without any complications. Blue Crystal also identified certain goals that were ‘future state’ and focused on successfully executing the first migration and saving the data from potential loss.
Blue Crystal worked with the Client’s team, to change all the pipelines for deployment, set up S3 for managing data files, and organise the routing, network management, and security—making sure the data was protected from external access.
The end result was a sophisticated environment capable of safely managing the transition.
Blue Crystal Solutions used extensive AWS services to operate this clean transition and mitigate risk, including:
- Automation and Infrastructure as Code using Terraform
- Implement a pilot environment including VPN and Active Directory
- Re-host: Lift-and-shift of older Linux instances to EC2 using AWS Application Migration Service (CloudEndure)
- Re-platform: Migration of Oracle Enterprise Edition to Standard Edition on RDS using DataPump and Postgres to Aurora
- Synchronise files from on-premise using Synology NAS with S3 integration
- Re-architect websites to use Amazon Certificate Manager, CloudFront, Application Load Balancers, and S3, including Lambda@Edge to replace Apache rewrite rules
- Provide automated database clones using the AWS CLI, including Route53 management
Freedom to focus on high value tasks
After Blue Crystal completed the migration—successfully maintaining all the data from the system and giving our client control of their stack again—they began to experience other ongoing benefits as well. With the lower-level services handled by AWS, the team can now focus on the application layer and the product sets they need, rather than spending time on admin-heavy operating tasks.
The client’s environment has since been completely upgraded onto a more sustainable level using containers with AWS Fargate, giving the team a platform to keep improving the way they build and deploy applications. Blue Crystal’s solution allowed them to take control and carry out change in an incremental way. And as they continue this evolution, they have ongoing access to the knowledgeable team at Blue Crystal, who can help them handle any roadblocks in the future.
An ongoing partnership
Just as HES continued to work with Blue Crystal, our Client engaged Blue Crystals’ services since the migration was completed in 2021. The next step in their steady system transformation is improving hosting costs, which will generate some more tangible outcomes for the organisation. Our Client intends to continue their support relationship with Blue Crystal for the foreseeable future.
“We’ve really appreciated the depth of knowledge and communication with Blue Crystal around migrating and incrementally improving our sustainability in AWS.” Digital Program Manager.
About Blue Crystal Solutions
Blue Crystal provides database, cloud, application, and infrastructure services across Australia. They work to empower organisations with the right IT and tech infrastructure to keep data safe, efficient, and geared towards growth.
“Blue Crystal’s approach is always to help us solve the problem in a way that works for us. They’re not dogmatic about how things should be done, they’re very customer focused—it’s about what outcomes are achievable within our constraints.” – Digital Program Manager.
