Eli, KHIT’s lead software developer, writes about RPAs.
As mentioned in our recent update blog, KHIT has recently implemented new technologies to improve our RPAs. You can read the basics about it in our 2021 tech update — basically, we transitioned our software to implement automations purely with Python and Selenium, and use Google Cloud Platform to host those automations. But for those more interested in the tech and how it’s implemented, I’ve decided to really dive into our automation process so we can show off what our RPAs are really capable of.
[For my fellow nerds out there, all the code in this blog (and more) can be found at my github page.]
Selenium & Python
For most of our clients, the task to automate begins with logging into a website that hosts a health record database that contains the information they need. To do this automatically, we use Selenium implemented with Python. Selenium is a sophisticated software that allows a regular programmer (me) to implement browser operations the way any person would. For this blog, I give an overview of navigating to a website, logging in, filling out forms, downloading and modifying those files, uploading them to Google Drive, and finally emailing notifications to clients once the process is complete. This is just one example of many types of automations we can do, but it should provide a nice overview.
To keep this short and simple, I’ll skip many of the technical details and challenges involved in the process. One such detail is setting up Selenium in the first place. All you need to know is that some outside files and programs are needed, and if you’re really interested I would just check out Selenium’s website.
Regardless, once setup is complete, the first step is to navigate to the site we need and log in using my account. This is done with the following Python code:
# login with open('src/config/login.yml', 'r') as yml: login = yaml.safe_load(yml) usr = login['fremont'] pwd = login['pwd'] driver.find_element_by_xpath('/html/body/form/div[3]/div[2]/div/div/div[1]/div/div[1]/div[1]/input').send_keys(usr) driver.find_element_by_xpath('/html/body/form/div[3]/div[2]/div/div/div[1]/div/div[1]/div[2]/div/input')\ .send_keys(pwd) driver.find_element_by_xpath('/html/body/form/div[3]/div[2]/div/div/div[1]/div/input[7]').click()
In this code block, I’m using a Selenium driver (the thing that mimics the browser) to find elements in the website’s HTML code and manipulate them either by sending them information or clicking them. This is done using XPaths, which are expressions that give us an easy way to traverse elements and attributes in HTML code. Once an element is found, we can use Selenium’s send_keys
function to type into the element, if applicable, or simply click on it. In this example, I fill out the username and password fields and then click the login button.
Once the driver is logged in, it navigates to and downloads a few different reports needed for this RPA. This is done using the same methods as the code block above, albeit with a few small technological differences that aren’t really worth getting into. After the reports are downloaded, I use some native Python libraries to rename them and place them in the appropriate directory. If modifications to the reports need to be made, such as merging or deleting unnecessary columns, the pandas
Python module is used. And violà, the Selenium part of the RPA is complete.
GSuite Operations
Since KHIT has partnered with Google, we have access to Google Cloud Platform (GCP) and all of its services. For RPAs, we make extensive use of Google’s service account feature. From their docs: “A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs.” In other words, service accounts make it possible to handle GSuite operations programmatically. We use our service account to upload files/folders to Google Drive and send email notifications through Gmail.
To upload files to Drive, the following code is used:
def service_act_login(): """ Log into gmail service account. """ SCOPES = ['https://www.googleapis.com/auth/drive.metadata', 'https://www.googleapis.com/auth/drive.file', 'https://www.googleapis.com/auth/drive', ] SERVICE_ACCOUNT_FILE = 'src/config/service_key.json' credentials = service_account.Credentials.from_service_account_file( SERVICE_ACCOUNT_FILE, scopes=SCOPES) service = build('drive', 'v3', credentials=credentials) return service def upload_file(filename, folder_id): print('Uploading file to drive...', end=' ') service = service_act_login() file_metadata = { 'name': filename.split('/')[-1], 'parents': [folder_id] } media = MediaFileUpload(filename, resumable=True) service.files().create(body=file_metadata, media_body=media, fields='id').execute() print('File uploaded.')upload_folder(file, '<some google driver folder id>')
This part can get complicated, but all this code does is use the service account’s unique JSON key to authenticate and login our account on Google’s end, then uploads the report we downloaded earlier into the folder we want. As long as your service account has proper permissions, this should work like a charm.
Likewise, we use the same service account to handle sending emails via Gmail API:
def send_gmail(email_to, subject, content): service = service_act_login() signature = '--\nSome signature' content = content + signature message = create_message(EMAIL_FROM, email_to, subject, content) sent = send_message(service, 'me', message) return sentemail_body = 'Some message' send_gmail('[email protected]', 'KHIT Report Notification', email_body
As you can see, this is very similar to the code for uploading a file to Drive. The service account is logged in and authenticated and the email is sent using that account, thanks to Google’s Python API library.
Hosting The RPA
GCP not only allows use of a dynamic service account, but also can be used to host our RPAs. GCP provides a way to implement RPAs remotely so they can run at any time, on any schedule: Linux virtual machines (VMs). VMs are emulations of computer systems designed to provide the functionality of a physical computer without the hassle of hosting a physical server. In our case, GCP has several facilities full of servers containing these VMs, which we can access via Google Cloud.
Once we have access to a VM in GCP, we can simply use clone our project into it (using git). Once the project has been test on the VM and works correctly, it’s time to schedule the RPAs. To do so, we use crontab
, a feature of Linux that allows us to run shell scripts on any schedule. Here is an example of what one looks like:
This looks more complicated than it actually is — the first five numbers indicate the minute, hour, day, month, and year the script will run, with the *
meaning every minute, hour, etc. After the schedule is defined, the console command to run is next — for our automations, I use shell scripts to execute Python scripts (and don’t worry about the stuff after the two arrows, that’s just for logging purposes). Here’s one of the shells scripts we use to run the RPAs:
All the shell script does is go into the RPA project directory, activate the project’s virtual environment, and runs the Python automation. Once the crontab is scheduled to run our scripts, we can just let the Linux VM do its thing.
And that’s it! With the exception of some technical details, that’s how KHIT RPAs are made.