I came, I saw, I borrowed the book. Do your students enjoy visiting the library just as Julius Caesar enjoyed victories? To receive such feedback from all students, including freshmen, your library must serve them without a long wait in line. In real life, a routine check, such as for permissions to borrow specific medical books, takes significant time and effort from the library personnel.
Synchronizing data between a student information system and a library management application is a common task. Recently, our team automated the data integration between Ellucian Colleague and Ex Libris Alma. Soon, Ellucian invited the ABCloudz team to work on a similar project with different challenges and infrastructure. Understanding the importance of library integrations, our team created reusable data pipelines that we can quickly and easily tailor to the needs of every customer.
In this blog post, we focus on creating reusable Data Connect pipelines that extract, transform, and load student data into the third-party library management system. However, our expertise covers many other Ellucian-based integrations. These integrations include automation and modernization projects for higher education institutions adopting Ellucian Banner SaaS. You can read more about integrations for parking, student code of conduct, and alumni management systems.
Are you looking for a reliable partner to bring your automation ideas to life?
See how the ABCloudz team helps other higher education institutions and learn more about our experience with Ellucian-based projects.
Understanding the integration challenges
The goal of this project is to synchronize student data with the library management application. To manage borrowing rights, the library system requires such personal details as name, email, address, and phone number. In addition to this information, we must keep in sync such program information as academic programs and disciplines, majors and minors, and dates of enrollment and graduation.
This information might change over time, but the most impactful change happens during the enrollment of new students. As a result, the ABCloudz team proposed to create three Data Connect pipelines and implement the change data capture (CDC) technique.
The first pipeline delivers a full student data record in response to any term-enrollment update. The second one tracks changes to the personal student data. The third pipeline tracks changes to the university-related data, such as programs or graduation dates. These two pipelines migrate only new, updated, or deleted records to the target database. With this CDC technique, universities can reflect changes as they occur or periodically, on a scheduled basis.
However, let us first look at the initial state of the customer’s data flow.
Current state architecture
The following image shows the current state of the customer’s data architecture.
At the first step, administrators manually run SQL scripts to extract data from the on-premises Banner database. Next, they format the extracted CSV files into a compatible format for the library management system. To complete the standard extract, transform, and load (ETL) flow, administrators upload these formatted files to the SFTP server.
As you can see, administrators have to deal with sensitive personal data. Therefore, only authorized users can run these data extraction scripts. In turn, data differences become sufficient when these authorized users are out of the office or busy with other tasks.
Creating reusable data pipelines
The ABCloudz team automated this ETL process by implementing three Data Connect pipelines. This effort was part of a larger modernization project, where our customer migrated from on-premises Banner database to the Ellucian Banner SaaS data environment.
To ensure that we can use these pipelines in other projects, ABCloudz created a configuration file for each pipeline. This file includes such settings as the Amazon S3 bucket, folder, access key, secret access key, or SFTP server credentials. As a result, the pipelines don’t include any hard-coded credentials, and we can securely share them with other institutions.
Of course, the data transformation part is specific to each project. Updating this part of the pipeline still requires attention before using this proven solution in other integrations.
Future state architecture
The following image shows the future state architecture of this automated integration.
We start by creating a change-notification subscription for the required Banner APIs. Once the subscription is active, the pipelines use the Ethos Consume Change Notifications service to retrieve new and updated records in the Ellucian Banner SaaS database. Following, we add the access credentials to the pipeline settings file. Next, we run the first pipeline that migrates all student data to the library management system. At the last step, we ensure that the second and third pipelines run every night to migrate updated data records when the database load is far from its peak.
Administrators can run these pipelines when needed. For example, when a scheduled pipeline fails because of a connection issue, the administrator receives an error log and can start the pipeline again. Also, administrators don’t directly interact with personal data when running these pipelines. Because of that face, more users received permissions to run data migration pipelines, ensuring that the data in both databases is kept in sync.
Benefits and results
By completing this project, we ensured that students get access to library systems from the first day. This access is key to many programs within the university. By completing this project, we ensured that students get access to library systems from the first day. This access is key to many programs within the university. Also, not only can the students access the library building, but they can also use the library-associated online websites and applications.
Also, this automated solution ensures that the library has information about borrowing privileges for each student. For books that require authorization, students don’t need to prove that they are eligible to borrow them. They come, see, and take the book home. As easy as it was for Julius Caesar to win the Battle of Alesia.
The final benefit relates to our future customers. We can customize this proven solution to expedite your data integration projects.
Streamline your data integration projects with trusted experts
Ellucian trusts us with the delivery of many integrations for Banner SaaS, and this is for the reason. Take a moment to read more about the Ellucian-based projects that our team completed. You can rest assured that the ABCloudz experts know how to build Ellucian Banner SaaS integrations of any complexity. We have a successful track record with hundreds of higher education institutions, and your institution can continue this list of success stories. Don’t hesitate and contact ABCloudz today!