Website Automation for Complete Noobs Part 2

In the first article in the series, we installed Python & Selenium and used them to write a script that opens a browser. In today’s article, we’ll look at how to do some basic inspection of web pages and modify our script to interact with them.

Selenium can do most of the same things you can do with web pages–namely, clicking & typing. But, just like you read and interpret what you see to know what to do, you need to tell your script how to find and do the things you want it to.

The easiest way to do this is to use the “inspect” functionality that exists in most modern browsers to help find identifiers you can use in your code. Let’s try it out with everybody’s ol’ buddy Google. Browse to google.com, right-click the search box, and pick Inspect.

This will take you to the selected element in the web page’s code. This is the part where you start to get some choices. Selenium has a lot of different ways to find elements, so you need to know about what you can use and compare it to what’s available. Usually the most reliable way to get an element is by using its id.

Usually, using an element’s id property along with Selenium’s find_element_by_id method is the most reliable way to find the element, but we don’t see an id when we look at Google’s search box.

We can see a name property with the value q, though, and there’s a find_element_by_name in Selenium, too; so let’s use that! With the element selected, you can use Selenium’s send_keys method to simulate typing. Add the following lines to your script:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')
search = driver.find_element_by_name('q')
search.send_keys('test search')

When you run this script, the browser will open and send the specified text to the search box. Now we need to submit the search. Google has a search button, so we can repeat the steps to find the button and click it:

button = driver.find_element_by_name('btnK')
button.click()

But, that’s not really how you search with Google, right? No–you type your search and press enter. To simulate this, you can use the submit function on the search element:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')
search = driver.find_element_by_name('q')
search.send_keys('test search')
search.submit()

Run the script again, and you’ll see your search submitted and search results displayed. Cool stuff! Let’s do a quick review of all the tools we have now:

  • Browse to sites using webdriver.get
  • Find elements using webdriver.find_element methods
  • Enter text with webdriver.send_keys
  • Click things with webdriver.click
  • Submit forms–like you would do by pressing enter on your keyboard after entering data–with webdriver.submit

Trackpad Disabled After Sleep/Suspend Following Upgrade to Ubuntu 20.04

Photo by Andrea Piacquadio from Pexels

As you may have guessed by my previous article, I recently upgraded to Ubuntu 20.04. Since the upgrade, I’ve had an issue where my laptop’s trackpad is disabled until I reboot–which is very annoying.

Today I set out to tackle the problem and found that this has been a recurring problem since at least 16.04, and it actually seems more surprising that I haven’t seen the issue until now! Luckily, there’s an easy solution.

I’m using a T440s, and this article about fixing the trackpad on a T450s was perfect. In this article, I’ll walk through my steps for confirming and fixing the issue.

First, I wanted to assert that I could reproduce the problem by performing the following steps:

  1. Reboot
  2. Confirm trackpad works
  3. Close laptop lid
  4. Open laptop lid
  5. Confirm trackpad is no longer working

Next, per the referenced article, I ran the following two commands to verify that they would re-enable a disabled trackpad:

$ sudo modprobe -r psmouse
$ sudo modprobe psmouse

Yay it worked! So, now I could put the permanent solution into place. I modified /etc/default/grub so that the GRUB_CMDLINE_LINUX_DEFAULT property included psmouse.synaptics_intertouch=0, then ran update-grub. These two tasks can be accomplished by running the following:

$ sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"/GRUB_CMDLINE_LINUX_DEFAULT="\1 psmouse.synaptics_intertouch=0"/g' /etc/default/grub
$ sudo update-grub

That’s it! I rebooted, repeated my repro steps, and confirmed that the trackpad was no longer disabled. Wonderful!

Website Automation For Complete Noobs Part 1

Image by mohamed Hassan from Pixabay

Have you ever been doing something on the internet and been like, “Ugh. I wish there was a way to do this automatically.” Well, most of the time there is: you can build some website automation using Python and Selenium. In part 1 of this series, I’ll walk you through installing Python and Selenium, and we’ll write a short script that opens a browser.

Install Python

For folks that don’t know, “Python is a scripting language that lets you work quickly and integrate systems more effectively.” (python.org) That means you write code in files and run the files without having to compile and deploy. In order to run the files, though, you need to install Python on your computer.

One way to do this is to download the installer from the official Python website, here. However, I prefer to use an application called PyCharm. PyCharm is an IDE (integrated development environment) available for Windows, Linux, and Mac that gives you lots of nice helpers when writing scripts, and there’s a very nice free community edition that’s just perfect for people like you who are just getting started.

So that’s step 1 for us today: download and install PyCharm Community Edition. Installation is pretty straightforward; you can just Next through everything and accept the defaults.

Once the install is complete, you can launch PyCharm. The first time you run it, PyCharm will ask you about which theme you prefer. Pick what you like and click Skip Remaining and Set Defaults.

Create Project

PyCharm is ready to go, but we need to do a little more setup before we’re ready to write our code. Begin by creating a new project from the PyCharm launch screen.

Give your project a name to specify where it will live on your computer, and click Create. If you don’t already have Python installed (the actual programming language) at this point, PyCharm will install it for you.

Install Selenium + Chromedriver

Selenium WebDriver is what we’ll use to launch and control the browser in our script. In order to use it, we need to install the libraries. We can install it directly from PyCharm by running a command in the terminal.

Open the terminal by clicking the button.

Now run the following command:

pip install selenium

Selenium also needs an extra application to help control the browser. In this example, I want to use Chrome as my browser, so I need to download ChromeDriver from here. When you visit the ChromeDriver page, it offers a few different versions. You need to pick the version based on the version of Chrome you have installed. This can be found in Chrome’s About page.

Once you determine which version you need, download the Zip file for your OS and extract chromedriver.exe to your project directory. You can verify that it’s in the right spot because it’ll be visible in PyCharm.

Write Script

Flip up those hoodies because it’s time to code. Add a new Python file to your project by right-clicking the project folder.

Add the following code to your script:

from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')

The first line tells our script that webdriver comes from the Selenium library that we installed. Then, it creates a variable named driver (but we could name it anything) that’s a Selenium WebDriver object using the ChromeDriver executable we downloaded. The third line uses driver to browse to http://www.google.com. Nothing too crazy, right?

Run Script

Go back to your terminal and run your script like this:

python <your-script>.py

When you press enter, a Chrome browser should open and navigate to the URL specified in your script. Chrome also gives an indicator that the browser is being controlled by automation software.

Pretty cool, right? There’s not much to experiment with this script. You can change the URL or add more driver.get lines to visit multiple sites. In the next article in the series, we’ll look at how to use WebDriver to interact with web sites and do things.

Angular & Firebase for Fast, Easy Database Apps

Photo by Campaign Creators on Unsplash

I recently started working on a new personal project that needed a database, and I just wanted something quick & easy. I remembered a colleague mentioning Firebase in the past and decided to take a look. After playing with it for a few days, I really dig it. I love that you can set up a database for free within a few minutes, and access it directly from the frontend without having to introduce a separate API layer.

In this article, I’ll walk through steps required to create a new Angular application and connect it to a Firebase Cloudstore database. You’ll need to sign up with Firebase to create the database, but it’s free and painless. Let’s get to it!

We’ll begin by creating a new application using the Angular CLI:

$ ng new firebase-app
$ cd firebase-app

Next, we’ll create our Firebase Cloudstore database. Go to firebase.com, and create an account or sign in. Add a new project and Cloudstore database in test mode. Finally, add a web app to the project. When you do this, you’ll be presented with instructions for adding the Firebase SDK to your app, as shown below.


Copy/paste the configuration details into our applicaton’s app.module.ts file. (Don’t worry about putting it in the right spot–we’ll do that in a minute. Just get it in there so you have it.) Now it’s time to install the Firebase and the Angular Firebase module:

$ npm -i firebase @angular/fire

When you install @angular/fire a browser will open for you to authenticate with Firebase, and you’ll be given an authorization code. Copy/paste the code back to the command prompt to finish installing the module.

With the module installed, we complete the necessary changes in app.module.ts:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { AngularFireModule } from '@angular/fire';
import { AngularFirestoreModule } from '@angular/fire/firestore'

import { AppComponent } from './app.component';
import { DemoComponent } from './demo/demo.component';

// Your web app's Firebase configuration
var firebaseConfig = {
  apiKey: "AIzaSyAx2cAfq9Pj3EzavXLkNc6_F9zWCyIayY4",
  authDomain: "ap-sample-1218e.firebaseapp.com",
  databaseURL: "https://ap-sample-1218e.firebaseio.com",
  projectId: "ap-sample-1218e",
  storageBucket: "ap-sample-1218e.appspot.com",
  messagingSenderId: "200601572991",
  appId: "1:200601572991:web:a335d1e106542870a9914a"
};
  
@NgModule({
  declarations: [
    AppComponent,
    DemoComponent
  ],
  imports: [
    BrowserModule,
    // Initialize Firebase
    AngularFireModule.initializeApp(firebaseConfig),
    AngularFirestoreModule,
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

At this point, we’re actually done with setup and fully functional, but let’s add a new component to demonstrate its usage:

$ ng g c demo

Open demo.component.ts, and add the following:

import { Component, OnInit } from '@angular/core';
import { AngularFirestore } from '@angular/fire/firestore';
import { Observable } from 'rxjs';

@Component({
  selector: 'app-demo',
  templateUrl: './demo.component.html',
  styleUrls: ['./demo.component.css']
})
export class DemoComponent implements OnInit {

  items: Observable<any[]>;
  
  constructor(
    private firestore: AngularFirestore
  ) { 
    this.items = this.firestore.collection('items').valueChanges();
  }

  ngOnInit(): void { }

  saveItem(value: string): Promise<any> {
    return this.firestore.collection('items').add({
      value: value
    });
  }
}

Add some simple markup to add & display items in demo.component.html:

<input #input type="text">
<button (click)="saveItem(input.value)">Add</button>

<ul>
    <li *ngFor="let item of items | async">
        {{item.value}}
    </li>
</ul>

Finally, remove boilerplate code from app.component.html and replace it with the demo component:

<app-demo></app-demo>

Now run the app. When values are submitted, they’re saved to our Firebase Cloudstore database, and the page will update in realtime as new records are added. Not bad for a few minutes of work!

$ ng serve --open

Make a Trello Clone in 15 Minutes

In the past weeks, I’ve written about how to make a drag & drop list using Angular CDK and how to enable dragging & dropping between multiple lists. Trello is the app that first comes to mind when I think of how this can be used to create a great experience.

At its core, a Trello board has just three pieces of functionality: create lists, create cards within the lists, and reorganize cards within and between the lists. The articles above give us all the tools we need to do this quickly ourselves, so let’s do it!

We’ll use the Angular CLI to do the following:

  1. Create a new app
  2. Add a list component
  3. Add a board component
  4. Make it pretty

Create New App

This first step’s pretty easy. We’re just going to use ng to create a new app and install the Angular CDK.

$ ng new lists-app --defaults
$ cd lists-app
$ npm install @angular/cdk

Do a quick check to make sure we’re starting in a good state:

$ ng serve --open

Add List Component

Now we’ll make our list component using ng generate component.

$ ng g c list

Hop into the code and make three changes. First, we need to import the DragDrop and Forms modules in src/app/app.module.ts:

import { BrowserModule } from '@angular/platform-browser';
import { DragDropModule } from '@angular/cdk/drag-drop';
import { FormsModule } from '@angular/forms';
import { NgModule } from '@angular/core';

import { AppComponent } from './app.component';
import { ListComponent } from './list/list.component';

@NgModule({
  declarations: [
    AppComponent,
    ListComponent
  ],
  imports: [
    BrowserModule,
    DragDropModule,
    FormsModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

Next, update src/app/list/list.component.ts to support drag/drop and dynamic creation of items. We’ll add the items array to store our list items, a drop function to handle our drop/drop event, and an onSubmit function to add new items to our list.

import { Component, OnInit } from '@angular/core';
import { CdkDragDrop, moveItemInArray, transferArrayItem } from '@angular/cdk/drag-drop';
import { NgForm } from '@angular/forms';

@Component({
  selector: 'app-list',
  templateUrl: './list.component.html',
  styleUrls: ['./list.component.css']
})
export class ListComponent implements OnInit {

  items: string[] = [];

  constructor() { }

  ngOnInit(): void {
  }

  drop(event: CdkDragDrop<string[]>) {
    if (event.previousContainer === event.container) {
      moveItemInArray(
        event.container.data,
        event.previousIndex,
        event.currentIndex);
    } else {
      transferArrayItem(
        event.previousContainer.data,
        event.container.data,
        event.previousIndex,
        event.currentIndex);
    }
  }

  onSubmit(newItemForm: NgForm) {
    this.items.push(newItemForm.value.newItem);
    newItemForm.reset();
  }
}

The third step is to change our markup in src/app/list/list.component.html. This is just two parts, displaying the drag-&-droppable list items and accepting input for new items.

<div cdkDropList [cdkDropListData]="items" (cdkDropListDropped)="drop($event)">
    <div *ngFor="let item of items" cdkDrag>{{item}}</div>
</div>
<form #newItemForm="ngForm" (ngSubmit)="onSubmit(newItemForm)">
    <input name="newItem" ngModel type="text" placeholder="Enter a new item"><button type="submit">Add Item</button>
</form>

That’s it for our list component, but let’s make one more change to app/app.component.html so we can test. Replace its entire contents with the following:

<app-list></app-list>

Now let’s do another check-in. We should be able to add items to our list and move them around via drag & drop.

$ ng serve --open

Add Board Component

Once again, we look to ng generate component to create our board component.

$ ng g c board

Just like the list component allows for dynamic creation of list items, we want our board component to allow dynamic creation of lists. So, let’s modify src/app/board/board.component.ts to support this:

import { Component, OnInit } from '@angular/core';
import { ListComponent } from '../list/list.component';

@Component({
  selector: 'app-board',
  templateUrl: './board.component.html',
  styleUrls: ['./board.component.css']
})
export class BoardComponent implements OnInit {

  lists: ListComponent[] = [];

  constructor() { }

  ngOnInit(): void {
  }

  addList() {
    var newList = new ListComponent();
    this.lists.push(newList);
  }
}

And make the markup changes in src/app/board/board.component.html. One thing to note is the use of cdkDropListGroup. This makes all the lists connected and allows dragging & dropping between them.

<button (click)="addList()">Add List</button>
<div class="list-container" cdkDropListGroup>
    <app-list *ngFor="let list of lists"></app-list>
</div>

We’ll also modify src/app/board/board.component.css so that lists will be added horizontally.

.list-container {
    display: flex;
    flex-direction: row;
}

Finally, we’ll update app.component.html to use our board component instead of a single list:

<app-board></app-board>

Our board component is complete, so let’s do another check-in.

$ ng serve --open

Make It Pretty

At this point, the hard part’s done. We have our core functionality implemented. We can add lists, add items to the lists, and move the items around. Now let’s make it look nice!

Begin by installing Angular Material:

$ ng add @angular/material

Then import the MatCard, MatButton, and MatInput modules in src/app.module.ts:

import { BrowserModule } from '@angular/platform-browser';
import { DragDropModule } from '@angular/cdk/drag-drop';
import { FormsModule } from '@angular/forms';
import { MatButtonModule } from '@angular/material/button';
import { MatCardModule } from '@angular/material/card';
import { MatInputModule } from '@angular/material/input';
import { NgModule } from '@angular/core';

import { AppComponent } from './app.component';
import { ListComponent } from './list/list.component';
import { BoardComponent } from './board/board.component';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';

@NgModule({
  declarations: [
    AppComponent,
    ListComponent,
    BoardComponent
  ],
  imports: [
    BrowserModule,
    DragDropModule,
    FormsModule,
    BrowserAnimationsModule,
    MatButtonModule,
    MatCardModule,
    MatInputModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

Now we can use mat-card, mat-raised-button, and mat-form-field in src/app/list/list.component.html:

<div class="container">
    <div class="list" cdkDropList [cdkDropListData]="items" (cdkDropListDropped)="drop($event)">
        <mat-card class="list-item" *ngFor="let item of items" cdkDrag>
            <mat-card-content>
                <p>{{item}}</p>
            </mat-card-content>
        </mat-card>
    </div>
    <form #newItemForm="ngForm" (ngSubmit)="onSubmit(newItemForm)">
        <mat-form-field>
            <input matInput name="newItem" ngModel type="text" placeholder="Enter a new item">
        </mat-form-field>
        <button mat-raised-button type="submit" color="accent">Add Item</button>
    </form>
</div>

And we’ll add a little CSS to src/app/list/list.component.css:

.container {
    margin-right: 10px;
}

.container button {
    margin-left: 5px;
}

.list {
    padding: 10px;
    max-width: 100%;
    border: solid 1px #ccc;
    min-height: 60px;
    display: block;
    background: #fafafa;
    border-radius: 4px;
    overflow: hidden;
}

.cdk-drag-preview {
    box-sizing: border-box;
    border-radius: 4px;
    box-shadow: 0 5px 5px -3px rgba(0, 0, 0, 0.2), 0 8px 10px 1px rgba(0, 0, 0, 0.14), 0 3px 14px 2px rgba(0, 0, 0, 0.12);
}

.cdk-drag-placeholder {
    opacity: 0;
}

.cdk-drag-animating {
    transition: transform 250ms cubic-bezier(0, 0, 0.2, 1);
}

.list-item {
    margin: 5px;
}

.list-item:last-child {
    border: none;
}

Then we’ll do some similar things to src/app/board/board.component.html and board.component.css:

<button (click)="addList()" mat-raised-button color="primary">Add List</button>
<div class="list-container" cdkDropListGroup>
    <app-list *ngFor="let list of lists"></app-list>
</div>
button {
    margin: 10px;
}

.list-container {
    display: flex;
    flex-direction: row;
    margin: 10px;
}

Now we’ve got fancy buttons, some input animations, and shadows while dragging, and it looks quite nice!

That’s where we’ll leave it today. The code I wrote while constructing this post can be found here.

Custom Background in Microsoft Teams

Microsoft Teams has had the ability to blur your background for a while, but they just expanded this to allow the use of a background image instead of just the blur effect. Out of the box, you’re only allowed to choose from a selection of Microsoft-provided images, but support for custom images is there–you just have to know where to drop your images.

So, here it is. Copy your images to the following directory, and they’ll show up in Microsoft Teams when you enable background effects:

%APPDATA%\Microsoft\Teams\Backgrounds\Uploads

If the Backgrounds\Uploads path doesn’t exist on your computer, it might be because you haven’t used the background effects feature yet. Turn on your camera, pick a background image, and the folder should get created. Or just go create it yourself.

Microsoft’s images use 1920×1080, so I recommend following suit for best results.

Upgrade to Ubuntu 20.04

Image by rockiger from Pixabay

The newest LTS version of Ubuntu is scheduled to release on April 23. Once it’s released, upgrading is as easy as running a few commands. In this article, we’ll walk through the process of updating the current system and then performing the upgrade.

Before you upgrade, make sure your system is as up to date as possible. Do this by running the follwing commands:

sudo apt update 
sudo apt upgrade
sudo apt dist-upgrade
sudo apt autoremove

Now that you’ve confirmed you’re up to date, it’s time to do the upgrade.

sudo apt install update-manager-core
sudo do-release-upgrade

If you receive a No new release found message, it means the upgrade hasn’t been made available to you. It’s been noted that the upgrade path from 19.10 will not be enabled until a few days after release, and the upgrade from 18.04 LTS will not be enabled until a few days after the 20.04.1 release expected in late July. However, you can force the upgrade at your own risk by using the -d flag.

sudo do-release-upgrade -d