Category Archives: web scraping

Automate discount coupons with Azure Automation and Microsoft Flow

Some background…
We’ve been buying our groceries online for a few years now. We find it super convenient and it saves us a lot of time. I even created a PowerShell module for it some time ago!

There is one (very minor) annoyance with it though, remembering to use the discount coupons you get after you’ve bought groceries for a certain amount. These coupons or codes get’s sent out before your current order has been delivered which means that you cant add them for your next order (can’t reach checkout while you have an active order waiting for delivery).

This means I have to wait for my order to be delivered and then add to it at the checkout step for my next order, at which point I’ve forgot all about it and maybe even deleted/archived the e-mail containing the pdf-file with the coupon.

I thought of this as the perfect scenario to check out a (relatively) new service from Microsoft called Flow, the idea behind Flow is to make it simple to automate things without the need of writing any code, but that doesn’t mean you can’t do that as well 🙂

How to achieve this?
When building automation I usually try to write down the steps needed to achieve the “end-to-end automation”. In this case that would be:

  1. Make sure the e-mails containing the coupons can be found automatically
  2. Get the coupon from the e-mail moved somewhere where it can be accessed by a PowerShell runbook in Azure Automation
  3. Create a PowerShell function that can parse pdf-files so the code inside can be retrieved
  4. Create another PowerShell function that can post the code to the online grocery store
  5. Profit! 🙂

These steps have now been achieved, and here’s how I did it:

Fetching the E-mail and the attachments (Step 1 and 2)
This is amazingly simple using Microsoft Flow. After you’ve signed up and logged in, just go to “My Flows” and click “Create from template”. There are quite a few to pick from so the easiest way to achieve this is to use the search function at the top of page, since I’m using as my personal e-mail provider, and thought the simplest way to store the attachments was using blob storage, I simply searched for “outlook blob” and found these templates I could use:


In my case, the first one fits perfectly so that’s the one I chose as a starting point. Click on it, pick “choose this template” and first connect your Azure storage account (needs to be created in advance):


Then connect your e-mail account by logging in:


If everything worked, you can go on and press “Continue”



You’ll then arrive at the page where you can configure the different steps in your flow, and if you want to, add some conditions. After you’ve clicked “edit” on both steps and updated them they should look something like this:

As you can see, I changed the folder this flow should look in from “Inbox” to “Flow” to prevent it from harvesting all the attachments I receive. I can then simply add a mail rule to put the e-mails I want in that folder.

Same thing for the “Create file”-step, “mailattachments” should correspond to a container on your storage account.

That’s it for parsing the e-mails. If you would like to, you could also add a http request after these steps to trigger the runbook automatically (webhook) as soon as a new attachment has been saved to the blob storage, but in this case, I’ll just schedule that to run at a regular intervall.

Parsing the pdf-file and posting the discount code (step 3, 4 and 5!)
To be able to get text out of the pdf-file I used the iTextSharp library. Then wrap that up in a PowerShell function, which in it’s simplest form might look something like this:

(Code example found at:

Add-Type -Path "$PSScriptRoot\itextsharp.dll"

function Get-PdfText
    param (
        [Parameter(Mandatory = $true)]

    $Path = $PSCmdlet.GetUnresolvedProviderPathFromPSPath($Path)

        $reader = New-Object iTextSharp.text.pdf.pdfreader -ArgumentList $Path

    $stringBuilder = New-Object System.Text.StringBuilder

    for ($page = 1; $page -le $reader.NumberOfPages; $page++)
        $text = [iTextSharp.text.pdf.parser.PdfTextExtractor]::GetTextFromPage($reader, $page)
        $null = $stringBuilder.AppendLine($text) 


    return $stringBuilder.ToString()

I’ve also added a function called “Add-MatHemBonusCode” to my “Grocery shopping PowerShell module“, because that got to exist, right? 😉

Finally, it’s time to wrap those functions up in a runbook.

The runbook could look something like this (dont look at this as a runbook best practice template, it’s not 🙂 ):

# Load the credentials needed
$AzureCredential = Get-AutomationPSCredential -Name 'AzureCred'
$MatHemCredential = Get-AutomationPSCredential -Name 'MatHem'

# Log into to Azure
Add-AzureRmAccount -Credential $AzureCredential

# Set a few parameters and fetch the storage information
$ResourceGroupName = 'MyResourceGroup'
$StorageAccountName = 'MyStorageAccount'
$ContainerName = 'mailattachments'
$StorageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $ResourceGroupName -Name $StorageAccountName

$StorageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey[0].Value

$Blobs = Get-AzureStorageBlob -Context $StorageContext -Container $ContainerName

# Filter out the attachment needed for this specific flow, only
# needed if you run multiple flows that look at attachments in the
# same container
$TargetedBlobs = $Blobs | Where-Object { $_.Name -match '^kvitto|^bonus' }

foreach ($MatHemBlob in $TargetedBlobs) {

    if ($MatHemBlob.Name -match '^kvitto') {
        # These are not needed so let's just remove them
        Remove-AzureStorageBlob -Blob $MatHemBlob.Name -Context $StorageContext -Container $ContainerName -Force
    elseif ($MatHemBlob.Name -match '^bonus') {
        # These contain the actual bonus or discount codes, so lets download those
        $LocalFileName = [System.IO.Path]::GetTempFileName()
        Get-AzureStorageBlobContent -Blob $MatHemBlob.Name -Container $ContainerName -Context $StorageContext -Destination $LocalFileName -Force

        # Fetch the text from the file
        $BonusPdfText = (Get-PdfText -Path $LocalFileName) -split "`n"

        # parse out the code itself
        $BonusCode = ((($BonusPdfText -match '^Värdekod:') -split 'Värdekod: ')[1]).trim()

        # Connect to the online grocery store and post the code (+some error handling and notifications)
        if (-not $Global:MathemSession) {
            Connect-Mathem -Credential $MatHemCredential

        if ($BonusCode) {
            try {
                $Results = Add-MatHemBonusCode -BonusCode $BonusCode -ErrorAction Stop
            catch {
                if ($_.ToString() -like '*Bonuskoden har redan använts i en annan order*' ) {
                    Write-Warning "Bonus code $BonusCode have already been used. Cleaning up blob..."
                    Send-PushNotification -Message "Bonus code $BonusCode from $($MatHemBlob.Name) have already been used. I'm cleaning up the blob."
                    Remove-AzureStorageBlob -Blob $MatHemBlob.Name -Context $StorageContext -Container $ContainerName -Force
                elseif ($_.ToString() -like '*Felaktig bonuskod*' ) {
                    Write-Warning "Bonus code $BonusCode is invalid. Notifying master..."
                    Send-PushNotification -Message "The bonus code $BonusCode from $($MatHemBlob.Name) was invalid. Please take care of this for me!"
                    Remove-AzureStorageBlob -Blob $MatHemBlob.Name -Context $StorageContext -Container $ContainerName -Force
                else {
                    Write-Warning "Failed to add bonus code $BonusCode from $($MatHemBlob.Name). The error was $($_.ToString())"
                    Send-PushNotification -Message "Failed to add bonus code $BonusCode from $($MatHemBlob.Name). The error was: $($_.ToString())"

            Send-PushNotification -Message "Bonus code $BonusCode from $($MatHemBlob.Name) have been added with the response: $Results"

Time to schedule that in Azure Automation, and make sure all the modules needed are available for it when it runs! (I run this on a hybrid worker)

While I have had a few issues with Microsoft Flow along the way (it is still in preview after all), it seems like a really cool service. And since you can make a http request to a webhook in Azure Automation, and/or just integrate them through some other service like the blob storage example in this post, the possibilities are pretty much endless.

So, as always… Keep automating anything!

Brewing coffee with Azure Automation

Can you automate anything with Azure Automation?
While there are some limitations on what it can and cannot do, I thought I could have a bit of fun using some of the fairly new features in Azure Automation to show that even though the main purpose of this service is to automate management tasks in the cloud (and in your local datacenter using hybrid workers), since it’s built on PowerShell, there really isn’t that much you cannot do.

I’ve been trying to come up with a scenario that is a bit of fun, and at the same time shows how you can use features like hybrid workers and webhooks to overcome almost any obstacles you have when automating something that spans over different services and locations.

This is what I came up with:
Let’s say I’m on my way home from work, it’s autumn and the rain is pouring down. I feel tired, cold, and just want to come home and grab a nice cup of coffee. To detect if something is running out of resources (in this case me), and to trigger something that can fix it (in this case, coffee) is a pretty common scenario in IT.

And to simulate that a process like this might span over multiple services where some are in the cloud (in this case iCloud, twitter and a weather service), while others are in your local datacenter (or in this case the coffee brewer in my kitchen), we are going to run parts of the code in Azure Automation and parts of it on a hybrid worker, everything orchestrated with Azure Automation.

The steps involved, at a high level, will be:

  • Fetch the location of my phone through iCloud
  • Use that information to fetch the weather data at that location, and check if it’s going to rain
  • If rain was detected, start with sending out a tweet asking if I would like some coffee, if the reply is positive, brew some coffee.
  • If no reply to the tweet is detected, it will send out an e-mail with a link to a page where the coffee brewer can be started with a button (that calls a runbook through a webhook)

So let’s get started!

So, where am I? And how’s the weather?
I don’t work at the same place everyday, so I don’t want to hard code the location where the weather is checked. Wouldn’t it be great if I could just fetch this information dynamically somehow? Well, with Azure Automation and some PowerShell, I can.

Since I carry around a smartphone with a GPS all day long, I thought that would be a good source for location details, and since you can fetch your location information through iCloud when you have an iPhone, this was the method I chose to do it.

Disclaimer: This code is for educational purposes ONLY, I do not take any responsibility if you use this outside of the ToS for the different services utilized here.

So I started with creating a PowerShell-function that could fetch my phones location through iCloud, if you want to take a look at it, it’s available here.

We then need to fetch some weatherdata at that location, luckily, I’ve already built a function like that before, blog post available here.

So, I have the tools to fetch my current location and the weather at that location. But how do to use this in Azure Automation?

Importing custom modules in Azure Automation
This is actually really simple! You can import almost any PowerShell module into Azure Automation, as long as you zip it up in folder with the same name as your module file. So I took my two functions above and put them into a WebUtilities.psm1-file. I then put that file into a WebUtilities-folder, and finally zipped it all up as “”. If you want to learn more about how to create integration modules for Azure Automation, including creating an optional file containing information about a Azure Automation connection-variable, more information about that is available here.

We then need to import this into Azure Automation. The screenshots that follows are from the “classic portal”, but you can do this in the preview portal as well:

First find the automation account you want to use, go to assets, and then click “Import Module” at the bottom:


Browse to your zip-file and click open to select it and press “Complete” down in the right corner:

Azure Automation will then begin to import the module and extract the activities it contains, you can follow the process at the bottom of the page:

These functions are now available in our PowerShell Workflows and PowerShell runbooks. Neat huh?

(The custom modules you import will not, at the time I’m writing this, be pushed to your hybrid runbook workers automatically. The Azure Automation team is working on that though, so it will happen eventually. In the meantime, you need to do this yourself.)

Writing the code…
It is now time to use the functions and actually write the code needed to tie everything together. There are many cool new features regarding Azure Automation but one of my favorites are the PowerShell ISE AddOn the Azure Automation team is working on, if you work with Azure Automation I can’t recommend you to check out the GitHub repository for it enough, and ever since I did the build straight from the source it has been working pretty well considering it’s still a very early release.

This is how my setup looks (ISESteroids, another great product, is also used here):

In addition to enabling you to use all of the features of the PowerShell ISE (and ISESteroids if you use that), this AddOn enables you to for example; fetch your runbooks straight from Azure, upload changes, run the code locally with emulated activities, test the code in Azure, and manage your assets so they are available when you test the code locally.

The productivity boost you get from this in comparison to the text authoring and testing experience in the portal, at least in my experience, is huge. So go ahead and try it out!

So, back to the code itself. As stated above, the steps involved here will be:

  • Fetch the location of my phone through iCloud
  • Use that information to fetch the weather data at that location, and check if it’s going to rain
  • If rain was detected, start with sending out a tweet asking if I would like some coffee, if the reply is positive, brew some coffee.
  • If no reply to the tweet is detected, it will send out an e-mail with a link to a page where the coffee brewer can be started with a button (that calls a runbook through a webhook)

Regarding tweeting from PowerShell, I want to give full credit of that to Adam Bertram‘s MyTwitter-module, thank you Adam! 🙂

And since the PowerShell community is so awesome, this is a pretty common scenario aswell, you build a few functions of your own, and you find some from others. Just zip it up and import it in the same way as the above functions. To use the MyTwitter-module, you also need to add API keys, just follow Adam’s instructions and you’ll be fine!

The code for the runbook, which I haven’t put too much effort into since it’s mostly a proof of concept, looks like this (native PowerShell script runbook!):

# Fetch my mobile device name
$DeviceName = Get-AutomationVariable -Name 'MyDeviceName'

# Fetch my iCloud Credential
$iCloudCred = Get-AutomationPSCredential -Name 'iCloudCredential'

Write-Output 'Fetching device location...'

# Let's start with fetching my location details
$MyDeviceLocation = Get-AppleDeviceLocation -Credential $iCloudCred | Where-Object { $_.DeviceName -eq $DeviceName }

# Check if we got a lock
if (!$MyDeviceLocation) {
    # Sometimes it takes longer for the device to locate, let's wait and try again
    Start-Sleep -Seconds 60
    $MyDeviceLocation = Get-AppleDeviceLocation -Credential $iCloudCred | Where-Object { $_.DeviceName -eq $DeviceName }

    if (!$MyDeviceLocation) {
        throw "Failed to fetch the location of device $DeviceName"

Write-Output "The following data was fetched from the device:`nLong: $($MyDeviceLocation.Longitude)`nLat: $($MyDeviceLocation.Latitude)"

# Time to get a weather report for my location
$CurrentWeather = $MyDeviceLocation | Get-SMHIWeatherData | Where-Object { [datetime] $_.ForecastEndDate -lt (Get-Date).AddHours(2) }

if ($CurrentWeather.PrecipitationCategory -contains 'Rain') {

    Write-Output "Rain is predicted soon, I'm gonna ask if he wants a cup of coffee. Sending out a tweet..."

    $SourceTweetHandle = Get-AutomationVariable -Name 'SourceTweetHandle'
    $TargetTweetHandle = Get-AutomationVariable -Name 'TargetTweetHandle'

    $Tweet = Send-Tweet -Message "@$TargetTweetHandle I got a feeling you would you like some coffee. Want me to fix it for you?"

    $NoReply = $true

    $NrOfLoops = 0
    $MaxNrOfLoops = 20

    while ($NoReply -AND $NrOfLoops -lt $MaxNrOfLoops) {


        Remove-Variable ReplyTweet -ErrorAction SilentlyContinue
        $TweetTimeline = Get-TweetTimeline -Username $TargetTweetHandle -IncludeReplies -MaximumTweets 20

        if ($TweetTimeline.in_reply_to_status_id_str -contains $Tweet.id_str) {
            $ReplyTweet = $TweetTimeline | Where-Object -FilterScript { $_.in_reply_to_status_id_str -eq $Tweet.id_str -AND $_.user.screen_name -eq $TargetTweetHandle }

            # Make sure we got a reply
            if ($ReplyTweet) {
                Write-Output 'Got a reply!'
                $NoReply = $false
        else {
            Write-Output 'Waiting for a reply...'
            Start-Sleep -Seconds 60

    # Make sure we got a reply and didn't just time out
    if ($ReplyTweet) {
        $PositiveReply = Get-AutomationVariable -Name 'PositiveReplyRegex'

        if ($ReplyTweet.text -match $PositiveReply) {
            Write-Output 'The reply was positive. Sending confirmation tweet and starting coffee brewer!'

            $ConfirmationTweet = Send-Tweet "@$TargetTweetHandle Consider it done."

            $AzureCred = Get-AutomationPSCredential -Name 'JarvisCred'
            $null = Add-AzureAccount -Credential $AzureCred
            Select-AzureSubscription -SubscriptionName 'Main Azure Subscription'

            $JobInfo = Start-AzureAutomationRunbook -Name 'Start-CoffeeBrewer' -AutomationAccountName Jarvis -RunOn 'JarvisGroup'

            Write-Output "Runbook started on hybrid worker group. I'm done here!"
        else {
            Write-Output 'The reply was negative. Sending confirmation tweet.'
            $ConfirmationTweet = Send-Tweet "@$TargetTweetHandle Alright, I wont do it then..."
    else {

    Write-Output "No reply on tweet detected, let's send out an e-mail instead."

    $WepageLink = Get-AutomationVariable -Name 'StartCoffeeBrewerPage'

# Set the body
$body = @"
Since the weather seems to be bad at your current location, I thought you might feel a bit cold.<BR>
If you feel a nice cup of coffee would help, just follow <a href='$WepageLink'>this link</A> and press the button on the page and I'll start the coffee brewer for you!<BR>
Kind regards,<BR>
Jarvis, running in Azure Automation<BR>
PS. I tried to tweet you but didn't get a reply, so I sent you this e-mail instead. DS.

    $SMTPCred = Get-AutomationPSCredential -Name 'SMTPAuthCredential'

    $MailMessageParams = @{
        'To' = Get-AutomationVariable -Name 'MyEmailAddress'
        'From' = "Jarvis <$($SMTPCred.UserName)>"
        'Subject' = 'Would you like some coffee?'
        'Body' = $body
        'UseSsl' = $true
        'Port' = Get-AutomationVariable -Name 'SMTPServerPort'
        'SmtpServer' = Get-AutomationVariable -Name 'SMTPServer'
        'Credential' = $SMTPCred
        'BodyAsHtml' = $true

    Send-MailMessage @MailMessageParams

    Write-Output 'E-mail is sent.'
else {
    Write-Output 'Seems the weather is fine, you have to make your own coffee!'

If you have read some posts at this blog before, you probably know that I enjoy creating home automation scripts quite a lot, and I’ve named this little project Jarvis after the famous AI, the ‘JarvisGroup’ specified above (Start-AzureAutomationRunbook cmdlet) is the hybrid worker group that runs some of these scripts. If you want to learn more about hybrid runbook workers and how to deploy them, check out this link.

Currently, you can’t use webhooks to trigger runbooks on a hybrid worker, as a workaround, I have another runbook that uses the Start-AzureAutomationRunbook cmdlet to trigger it on the hybrid worker instead, the code of that looks like this:

workflow Start-CoffeeBrewerThroughAzure
    $AzureCred = Get-AutomationPSCredential -Name 'JarvisCred'
    Add-AzureAccount -Credential $AzureCred
    Select-AzureSubscription -SubscriptionName 'Main Azure Subscription'

    Start-AzureAutomationRunbook -Name 'Start-CoffeeBrewer' -AutomationAccountName Jarvis -RunOn 'JarvisGroup'

To add a webhook to that runbook, you need to be in the Azure Preview portal, when you open the runbook details you’ll see the icon for creating a webhook, it looks like this:

Click on it, select “Create a new webhook”:

This will get you to this page:

Fill out the details of your new webook, and don’t forget to copy the link before clicking OK!

Voila, you’ve created a webhook! If you want to get more information regarding webhooks, check out this link.

The final thing we need now is the code for starting the coffee brewer (Start-CoffeeBrewer), I’m using the Home Automation Module I’ve written to achieve this. The runbook code looks like this:

workflow Start-CoffeeBrewer
	$TelldusCred = Get-AutomationPSCredential -Name 'TelldusCred'
	$CoffeeBrewerDeviceID = Get-AutomationVariable -Name 'CoffeeBrewerDeviceID'
	InlineScript {
		Write-Output 'Connecting to Home Automation Service...'
		Connect-TelldusLive -Credential $using:TelldusCred
		Write-Output 'Turning on the coffee brewer...'
		Set-TDDevice -Action turnOn -DeviceID $using:CoffeeBrewerDeviceID

The module containing the Connect-TelldusLive and Set-TDDevice cmdlets are installed on the target hybrid worker since that’s where it will execute (and as stated above, the module won’t be pushed out to hybrid workers automatically from Azure Automation even if you have imported them there, but that will be fixed in the future).

So, we’re all set now…

But, does it all work?
Well, you’d obviously have to come by for coffee some time to see this for yourself, but yes, it actually does! 🙂

Here are some screenshots of the first runbook in action:

When it’s not raining, test ran in the portal:

When rain is detected, test ran from the PowerShell ISE AddOn:

Tweet screen shot:

And confirmation tweet:

You can also view the tweets at this link.

Mailmessage in phone:

The webpage form for starting a runbook through a webhook:

The code for that form with the token masked (be aware that posting a form like this on a public website without authentication is a MAJOR security risk depending on the runbook type, it’s only for demo purposes in this case):

<HEAD><TITLE>Coffee brewer start!</TITLE>
<form action='***************************' method='post'>
<FONT size ='4'>Press this button to start the coffee brewer:</FONT>
   <button type='Submit'>Brew Coffee</button>

And finally, a short video of the Coffee brewer being started through Azure Automation (including a fuzzy reflection of my tired self being mesmerized by the coffee (first cup of the day 😉 )):

Brewing coffee through Azure Automation from Anders Wahlqvist on Vimeo.

I hope this post have helped you to see how flexible Azure Automation actually is. PowerShell is truly versatile and a great “glue-language” to tie different services together. Even though using Azure for turning on a coffee brewer might be a bit overkill, if it’s possible to integrate a weather service, an iPhone, e-mail, twitter and a coffee brewer using it, it can probably manage your IT environment aswell, don’t you think? 🙂

As always, happy automating anything!

Ordering pizza with PowerShell (web scraping guide) – Part 2

So, we have created our Connect-OnlinePizza function and now have access to parts of the site that are only available when logged in. But how?

Remember the Invoke-WebRequest-cmdlet in the last post?
We specified a session variable in the Global scope, and that variable contains cookies and data to keep our session with the site consistent over multiple webrequests, and that’s what we’ll use in our next function, Get-MyOnlinePizzaAccountInfo.

First of all, we need to find what page holds the information we want. In this case, the page containing the account information was located at (it requires you to be logged in).

Make sure you ran the “Connect-OnlinePizza”-function first, that way the “$OnlinePizzaSession”-variable will be available and make it possible for us to reach this page and see the details of our account.

To fetch the page and load it into a variable you could do this (we save it to file because of the issue with the encoding name, see part 1 of this guide):

Invoke-WebRequest -Uri "" -Method Get -WebSession $Global:OnlinePizzaSession -OutFile .\dump.htm
$AccountInfo = Get-Content .\dump.htm -Encoding UTF8

If this worked, we should start looking in the “dump.htm”-file for where the name is, use Select-String or just open the file in notepad and search for it.

When you’ve found the line, you need to figure out how to trim away all the parts of the line that you don’t want. In my case, it looks like this:

<input type=text name=namn id=namn class="input-medium" maxlength=100 value="Anders Wahlqvist"/>

I’m by no means an expert in string manipulation or regex, so there is probably a better way of doing this, but I usually use the Split-operator to get the part I want. In this case we need to split the string after value=” and before “/> (or remove it). We also need to fetch this particular line from the sites html code.

Take a look at this line:

$AccountHolderName = ((($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`"")[1] -split "`"/>")[0]

That might look like a complete mess, but we’ll break it down! We first need to fetch the correct line that contains the name which we can do with:

$AccountInfo | Select-String -Pattern "name=namn id=namn"

We want to do the “splitting” on the results of that, and therefore we need to put parentheses around that command before we add the “-split” operator. So let’s split that up and see what happens:

PS> ($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`""
input type=text name=namn id=namn class="input-medium" maxlength=100 
Anders Wahlqvist"/>

As you can see, we get two tokens back, and we need the second one. This can easily be done by putting everything in another pair of parentheses and then just specify which one we want. Since the first one will be identified as 0, and the one we want 1, we will end up with this:

PS> (($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`"")[1]
Anders Wahlqvist"/>

To get rid of that last part, we could either use the "replace"-operator or do another split. In this case, the "replace"-operator might be the better choice, but in my experience the split-operator will provide a more robust and consistent result. The site might change and add something else after "/> on the same line, or there might be some white space that you didn't see, so let's just do another split, wrap that up in a new set of parentheses and
and select token 0 (first one), which will get us our original line:

$AccountHolderName = ((($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`"")[1] -split "`"/>")[0]

Hopefully this line doesn't seem as messy anymore 🙂

Now we repeat that for all the information we want, like this:

$Username = ((($AccountInfo | Select-String -Pattern "name=username id=username") -split "value=`"")[1] -split "`" />")[0]
$AccountHolderName = ((($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`"")[1] -split "`"/>")[0]
$AccountHolderMail = ((($AccountInfo | Select-String -Pattern "name=epost id=epost") -split "value=`"")[1] -split "`"/>")[0]
$AccountHolderStreet = ((($AccountInfo | Select-String -Pattern "name=adress1 id=adress1") -split "value=`"")[1] -split "`"/>")[0]
$AccountHolderPostalCode = ((($AccountInfo | Select-String -Pattern "name=postnummer id=postnummer") -split "value=`"")[1] -split "`"/>")[0]
$AccountHolderPhone = ((($AccountInfo | Select-String -Pattern "name=telefon id=telefon") -split "value=`"")[1] -split "`"/>")[0]

And finally, we create an object for it and send it to the pipeline:

$returnObject = New-Object System.Object
$returnObject | Add-Member -Type NoteProperty -Name Username -Value $Username
$returnObject | Add-Member -Type NoteProperty -Name Name -Value $AccountHolderName
$returnObject | Add-Member -Type NoteProperty -Name Email -Value $AccountHolderMail
$returnObject | Add-Member -Type NoteProperty -Name Address -Value $AccountHolderStreet
$returnObject | Add-Member -Type NoteProperty -Name PostalCode -Value $AccountHolderPostalCode
$returnObject | Add-Member -Type NoteProperty -Name Phone -Value $AccountHolderPhone

Write-Output $returnObject

So far so good, time to wrap this up in a function, we've already looked at that in the last post, so I'll just add the complete code here:

function Get-MyOnlinePizzaAccountInfo

    BEGIN {
        if ($OnlinePizzaSession -eq $null) {
            Write-Error "You must first connect using the Connect-OnlinePizza cmdlet"


        Invoke-WebRequest -Uri "" -Method Get -WebSession $Global:OnlinePizzaSession -OutFile .\dump.htm

        $AccountInfo = Get-Content .\dump.htm -Encoding UTF8

        Remove-Item .\dump.htm -Force -Confirm:$false -ErrorAction SilentlyContinue

        $Username = ((($AccountInfo | Select-String -Pattern "name=username id=username") -split "value=`"")[1] -split "`" />")[0]
        $AccountHolderName = ((($AccountInfo | Select-String -Pattern "name=namn id=namn") -split "value=`"")[1] -split "`"/>")[0]
        $AccountHolderMail = ((($AccountInfo | Select-String -Pattern "name=epost id=epost") -split "value=`"")[1] -split "`"/>")[0]
        $AccountHolderStreet = ((($AccountInfo | Select-String -Pattern "name=adress1 id=adress1") -split "value=`"")[1] -split "`"/>")[0]
        $AccountHolderPostalCode = ((($AccountInfo | Select-String -Pattern "name=postnummer id=postnummer") -split "value=`"")[1] -split "`"/>")[0]
        $AccountHolderPhone = ((($AccountInfo | Select-String -Pattern "name=telefon id=telefon") -split "value=`"")[1] -split "`"/>")[0]

        $returnObject = New-Object System.Object
        $returnObject | Add-Member -Type NoteProperty -Name Username -Value $Username
        $returnObject | Add-Member -Type NoteProperty -Name Name -Value $AccountHolderName
        $returnObject | Add-Member -Type NoteProperty -Name Email -Value $AccountHolderMail
        $returnObject | Add-Member -Type NoteProperty -Name Address -Value $AccountHolderStreet
        $returnObject | Add-Member -Type NoteProperty -Name PostalCode -Value $AccountHolderPostalCode
        $returnObject | Add-Member -Type NoteProperty -Name Phone -Value $AccountHolderPhone

        Write-Output $returnObject


    END { }

Take a look at line 7 through 10, here we check if there is a variable called "$OnlinePizzaSession" available, if not, the user running this function probably didn't run the "Connect-OnlinePizza"-function, and this function won't work. Therefor, we write an error and exit the function. This is a pretty good method to ensure that the functions are used correctly.

So, finally time for our last function!

Most parts of this function will be created more or less in the exact same way as the last one, so I'll just go through the differences.

First of all, we want these cmdlets to work together in a good way to give them that "module"-feeling 🙂

One way of doing that is to add pipeline support, but how?

Well, this function will return a list of restaurants based on our location, and the location is based on our postal code (zip code). If you check our last function we actually return a property value called "PostalCode" which would be perfect for pipelining, and it's really easy to do!

All we need is "ValueFromPipelineByPropertyName=$true" when declaring the parameter, like this:

          [int] $PostalCode)

And we need to verify that the property in object we output match the parameter name:

Also, as you can see, we are declaring the parameter data type as an int, this way, no one will give as a postal code with spaces in it. If we want to, we could also validate that it really is a postal code, but again, this guide is not as much about writing advanced functions in general but has more to do with web scraping, so we'll just let it be.

Let's look at the rest of this function:

function Get-PizzaRestaurant
          [int] $PostalCode)

    BEGIN {
        if ($OnlinePizzaSession -eq $null) {
            Write-Error "You must first connect using the Connect-OnlinePizza cmdlet"


        Invoke-WebRequest -Uri "$PostalCode" -Method Get -WebSession $Global:OnlinePizzaSession -OutFile .\dump.htm

        $ResturantList = ((Get-Content .\dump.htm) -join "`n") -split "<UL>" | select -Skip 1

        Remove-Item .\dump.htm -Force -Confirm:$false -ErrorAction SilentlyContinue

        foreach ($Restaurant in $ResturantList) {

            $RestaurantName = (($Restaurant -split "<h4>")[1] -split "</h4>")[0]

            if ($RestaurantName -eq '') {

            $RestaurantStreet = (($Restaurant -split "<address>")[1] -split "</address>")[0]
            $OpeningHoursDelivery = ((($Restaurant -split "Utkörning:</strong><br />")[1] -split "<br />")[0]).Trim()
            $OpeningHoursTakeAway = ((($Restaurant -split "Avhämtning:</strong><br />")[1] -split "<br />")[0]).Trim()
            $RestaurantLink = ((($Restaurant -split "meny")[0] -split "href=`"")[1] -split "`"")[0]

            $returnObject = New-Object System.Object
            $returnObject | Add-Member -Type NoteProperty -Name RestaurantName -Value $RestaurantName
            $returnObject | Add-Member -Type NoteProperty -Name RestaurantStreet -Value $RestaurantStreet
            $returnObject | Add-Member -Type NoteProperty -Name OpeningHoursDelivery -Value $OpeningHoursDelivery
            $returnObject | Add-Member -Type NoteProperty -Name OpeningHoursTakeAway -Value $OpeningHoursTakeAway
            $returnObject | Add-Member -Type NoteProperty -Name RestaurantLink -Value $RestaurantLink

            Write-Output $returnObject

            Remove-Variable RestaurantName, RestaurantStreet, OpeningHoursDelivery, OpeningHoursTakeAway, RestaurantLink -ErrorAction SilentlyContinue

    END { }

A few more comments might be needed here, if you look at line 19, we use the opposite of split, the join-operator. Why? Well, when looking at the html-code of the site the information is spanning over multiple lines, by joining on linefeeds (`n = linefeed) we can get all the information for each restaurant as "one part" instead of multiple lines, which helps a lot!

Also, at line 32 and 33, we call a method called Trim(), this method removes all leading and trailing white-space characters from the string we're working on.

Finally, at line 45 we remove all the variables to prevent them from being "reused" on the next iteration of the loop if the next restaurants data is different or missing. Clear-Variable would work perfectly here aswell.

And that's it!

We have now created functions to connect to a site, utilize functions that are only available when logged in and we have also made the functions work together in a nice way.

This is how they look in action:

Pretty neat, huh? 🙂

The code for all of these functions have been uploaded here.

I hope you enjoyed this little guide, and if you have any questions, feel free to ask them in the comments or drop me an e-mail!

And keep automating anything 🙂

Ordering pizza with PowerShell (web scraping guide) – Part 1

Since I’ve gotten some positive feedback regarding the web scrape related posts on this blog, I thought I should write a guide on how to build PowerShell functions that interacts with a website. To make it a bit more fun, I thought it should be about ordering pizza!

The site in question is a Swedish site called OnlinePizza.

Since the actual code won’t be used for anything serious, you will see a few shortcuts here and there, the goal is to give you an idea on how to do something similar, not to create the perfect pizza automation module. In fact, we’ll only create a few functions, one for logging in, one for checking our account information and one for listing restaurants. Hopefully, the process and steps we will go through when creating these functions will teach you the basics of web scraping. To create a whole module that can handle the complete process of ordering pizzas seems a bit overkill, but feel free to keep working on it if you want to 🙂

So with no further ado, let’s get to it…

Figuring out how the site works and logging in
The first thing we need to do is figuring out how to log in. The easiest way to do this is to open a web browser with some developer features, since I’ve used Chrome before I thought I could use Internet Explorer in this example.

Begin with locating the loginpage for the site, which in this example is: as shown below:
After you’ve browsed to that page and filled in your username/password, you press F12 (Ctrl+Shift+i if you’re using Chrome) and select the “Network”-tab and press the “Start capturing”-button.

Should look something like this if using IE:

Go back to the site and press the “login” button (“Logga in” in Swedish), and when it’s done, go back to the “network”-tab again and check the first request which is usually a Get or a Post request, in this case a Post-request:

Double click on the first row and select the “Request body”-tab. This is how the webrequest looked when it got sent to the webserver, so this is what we need to mimic from PowerShell:


This can be done in different ways, you could either basically download the site, try to manipulate the fields and then post the form, or create a hashtable with the required keys (username, password and action) and send that. The later is quickest since it only needs one request, so that’s what we’ll do here (this might not always work though).

To create the hashtable you can do this:

$Request = @{'username' = 'MyUsername'
             'password'= 'MyPassword'
             'action'= 'loggain'}

We now need to send it to the web server. The easiest way to do this is usually to use the “Invoke-WebRequest”-cmdlet that came with PowerShell v3, so that’s what we’ll do here. There are a few scenarios where this will give you issues, as is the case with this site, so we’ll need to use a workaround. When trying to download the site with the “Invoke-WebRequest”-cmdlet it will give you the error: “Invoke-WebRequest : ‘”UTF-8″‘ is not a supported encoding name.”

This leaves you two options as far as I know, you either skip the “Invoke-WebRequest”-cmdlet altogether and use the .Net WebClient Class instead, or you can fix this error by sending the output of the cmdlet to a file instead of the pipeline. We’ll do the latter here and save the .Net-method for another post.

Note: As stated above, this is mostly to give you and idea on how to create a “web scraping function” in PowerShell so we’ll do a few shortcuts. If this was a to become a serious module that would later be used in production, make sure the output-file is written to a place where the user will have write access and that it has a name that won’t damage (overwrite) anything.

The command itself is pretty simple, it looks like this:

Invoke-WebRequest -Uri "" -Method Post -Body $Request -SessionVariable Global:OnlinePizzaSession -OutFile .\dump.htm

I’ll break it down a bit so you’ll know what each parameter does:

  1. Uri – This specifies where to send the request. This should be the URL you saw in the screenshot of the “Request body” above, this can differ from the actual loginpage depending on the site.
  2. Method – This should match the method you saw in the loginrequest, in this case it was a “Post”-request.
  3. Body – This is what the request will actually contain, in our case the hashtable we created.
  4. SessionVariable – The variable we specify here (no leading $!) will contain cookies and other data needed to keep the session consistent through the rest of the commands we will run (this will for example keep us logged in). I’ve specified it in the “Global” scope since we want to use it together with other functions later on, and to make that work, it can’t be in the functions scope (since that will be gone before the next function will execute).
  5. OutFile – Our workaround. Specify a file where the output from the command (the html of the site) should be saved.

Alright, so the request is sent and we should now be logged in, you can verify this by looking in the “dump.htm”-file, it will usually contain a “You have been logged in!”-message of some sort. In this case that message is in Swedish though.

So, we have figured out how to log in, we now need to wrap a function around this, which will be our next step in this process!

Creating the function
To create this function we need to ask for a parameter, the only one we need in this case is the credential, which should be of the type PSCredential.

The code for defining the function name and the parameter looks like this:

function Connect-OnlinePizza
          [System.Management.Automation.PSCredential] $Credential)

As you can see, we also add the almost magical “cmdletbinding”-keyword aswell to get all the wonderful features that gives us. We also state the Credential-parameter as mandatory, and we define its data type, which will make the function ask for the credential in the same way as “Get-Credential” works if the user didn’t specify any.

We now need to place the credential in our request, which we can do this way:

$Username = $Credential.UserName
$Password = $Credential.GetNetworkCredential().Password

$Request = @{'username' = $Username
             'password'= $Password
             'action'= 'loggain'}

We then add our “Invoke-WebRequest”-command again:

Invoke-WebRequest -Uri "" -Method Post -Body $Request -SessionVariable Global:OnlinePizzaSession -OutFile .\dump.htm

And we should be logged in. When building something like this we should make sure though. This can easily be done by looking for that “You have been logged in!”-text in the dump.htm-file, for example with the “Select-String”-cmdlet.

And while we’re at it, why not delete the dump.htm-file to clean up a bit. That would look like:

$LoggedIn = Select-String -Path .\dump.htm -Pattern "inloggad som $Username" -Quiet

Remove-Item .\dump.htm -Force -Confirm:$false -ErrorAction SilentlyContinue

if ($LoggedIn) {
    Write-Verbose "You are now logged in!"
else {
    Write-Error "Login failed!"

And that’s it! When put together, the code looks like this:

function Connect-OnlinePizza
          [System.Management.Automation.PSCredential] $Credential)

    $Username = $Credential.UserName
    $Password = $Credential.GetNetworkCredential().Password

    $Request = @{'username' = $Username
                 'password'= $Password
                 'action'= 'loggain'}

    Invoke-WebRequest -Uri "" -Method Post -Body $Request -SessionVariable Global:OnlinePizzaSession -OutFile .\dump.htm

    $LoggedIn = Select-String -Path .\dump.htm -Pattern "inloggad som $Username" -Quiet

    Remove-Item .\dump.htm -Force -Confirm:$false -ErrorAction SilentlyContinue

    if ($LoggedIn) {
        Write-Verbose "You are now logged in!"
    else {
        Write-Error "Login failed!"

And this is how it looks in action:

That’s it for this post. In the next one, we’ll create the function for fetching our account information and one for getting a list of what restaurants are available in our location.

Checking TV Show schedules with PowerShell

Remember the post about buying groceries with PowerShell? One of the usage examples suggested that you could order some snacks when a tv show you like is having a season finale (or premiere for that matter!).

But how to check for this in an automated way? With PowerShell of course 🙂

I’ve written two advanced functions for doing this:

  • Get-TVShowNextAirDate retrieves a list of most tv shows that are currently airing and is listing their next airdate.
  • Get-TVShowAirDate retrieves all known airdates of the tv show specified (supports pipelining! 😉 )

Screenshot of these in action:

That’s all the tools you need to automate your popcorn order! 😉

The code can be viewed through this link (Updated 2016-10-06).

Happy automating anything!

Web scraping with PowerShell (Getting a package trace from a postal service)

Building an advanced function that can consume information on the web is pretty powerfull and I use it for all kinds of things.

In this post I will try to guide you through the process on how to build one for more or less any service, but the example will be the Swedish postal service.

I usually start with a web browser that has some developer features, for example Google Chrome. Go to the website and press Ctrl+Shift+i, select the “network tab” and enter whatever information you need to send to the service, in this case the ID of the package I want to trace.

In this example it should look like this: (I have chosen to use the English version of the website):

Press the submit button and look at the beginning of the network trace. You usually find a GET or POST request there, in this case it is a GET-request.
In this example it looks like this:

You can right click that row and select “Copy link address”, which in this case is “”.

Now open whatever PowerShell script environment you prefer, for example the PowerShell ISE. Start with sending the same request from PowerShell, that can be done by using Invoke-WebRequest (if you are using PowerShell v3 or higher). Start with putting a variable where “MyPackageId” is.

For example:

$Id = "MyPackageId"
$PackageTrace = Invoke-WebRequest -Uri "$Id" -UseBasicParsing

The “UseBasicParsing” switch is not mandatory here, but if you don’t need the html returned to be parsed into different objects it is a bit quicker.

We now need to parse the html-code stored in the “Content”-property to get what we want. This can be a bit time consuming, but with a little help from Chrome it gets easier.

Press the magnifier button and hover the mouse over parts of the site or parts of the HTML-code (if you select the “Elements-tab”) and you will soon find what part of the HTML code you need.

In this example the table-tag. Screenshot:

Now we need to do some string manipulation to get the parts we need properly formatted. In this case we want to split the HTML to get the parts between the start of the table and the end of it. What we have left is the rows with all the package events, find something that splits them up in to nice pieces, in this case the “tr class=” tag. The first of the rows that gets returned are some table information (containing a unique ID that might change) and the table columns, so we want to skip those. A oneliner that does all of this looks like:

$TraceItems = ((($PackageTrace.Content -split "<table class=`"PWP-moduleTable nttEventsTable`"")&#91;1&#93; -split "</TABLE>")[0]) -split "<tr class=" | Select-Object -Skip 2</code>

We can now loop through these items, parse them and build an object out of them. Each one of these items has three columns; a date, a location and a comment/tracking event. The columns are enclosed in the “TD”-tags so we can split them up at those.

When you have all the values we need we create the object and send it to the pipeline. Could look something like this:

foreach ($TraceItem in $TraceItems) {

    $EventDate = (($TraceItem -split "<td>")[1] -split "</td>")[0]
    $Location = (($TraceItem -split "<td>")[2] -split "</td>")[0]
    $Comment = (($TraceItem -split "<td>")[3] -split "</td>")[0]
    $PackageId = $Id

    $returnObject = New-Object System.Object
    $returnObject | Add-Member -Type NoteProperty -Name EventDate -Value $EventDate
    $returnObject | Add-Member -Type NoteProperty -Name Location -Value $Location
    $returnObject | Add-Member -Type NoteProperty -Name Comment -Value $Comment
    $returnObject | Add-Member -Type NoteProperty -Name Id -Value $PackageId

    Write-Output $returnObject

We now have “objectified” a website and made it useful in PowerShell! When we have come this far it’s a good idea to create an advanced function around it to make it really useful.

There are many good posts explaining how that is done, for example this one by Don Jones, so please refer to that if you need some help on getting started.

I have made a quick example of an advanced function out of the code written in this post which is available here.

This is how the function looks in PowerShell (MyPackageId actually seems to be a valid Id, but it looks a bit weird. The output in PowerShell matches the site though):

Good luck automating anything!

And if you want to learn more, checkout my webscrape guide in this post!

Buying groceries with PowerShell

Yes, really, buying groceries can also be done with PowerShell!

How? Well, you need to find someone who sells groceries through a website, and can deliver them to your house.

The rest is just webrequests!

This is not only (but maybee mostly… 😉 ) made as a part of my weird quest for doing strange things with PowerShell, but also a small part of my home automation project. I’ll give you some usage examples:

  • Let’s say the weather-cmdlet posted earlier gives you reports about rain, wind and cold weather for the coming week, how about adding some popcorn to your basket and other things you want for a nice movie-night at home?
  • Your favorite TV show is ending next week, add those popcorns again!
  • If the weather looks nice… order some things needed for a picnic?
  • Connect it to your voice control made with powershell and ask it to, for example, “add milk” when you take the last one from the fridge.
  • An anniversary is coming up (I have written a cmdlet for checking this, post is coming!), let’s order flowers! (Romance is all about spontaeous things, right? 😉 )
  • Find an intelligent fridge with an ethernet/WiFi connection and… well… maybee not 😉

Alright… Nothing lifechanging… But it still shows how versatile PowerShell can be, which is sort of my main point with all this!

This is a small example of how it looks in action:

The code can be downloaded through this link. (updated at 2016-10-06)

Since it will only be useful for Swedish users who live in cities where this service is available, the rest of you can just look at it as sample code if you want to do something similar.

Wake me up, when traffic calms down… (Home Automation)

Why do home automation with PowerShell?

There are certainly other solutions out there which are great, even excellent. For me personally, it’s mainly because it’s fun to be able to build parts of it by yourself, you learn a lot by doing it, but you can also base your tasks on almost any piece of information out there.

I’ll give you an example!

I’ve started to go to work after the traffic calms down in the morning, I have a pretty good idea of when this usually happens, but sometimes there is no traffic jams at all, and sometimes it’s completely hopeless.

Wouldn’t it be nice to be able to utilize live traffic information on the internet, and based on that trigger your wake up call? At least I thought so 🙂

First of all, try to find a provider for traffic information near you, and make sure you don’t break their ToS by fetching that information in a automatic way (ie. web scraping).

I won’t go into detail on how to build a webscrape-cmdlet right now, but I’ll show you how to use it when it’s done. (A guide to web scraping available here, here and here.)

This is the script I run every morning, I think the code comments will be enough to explain how it works:

# Import the Telldus module
Import-Module '.\Telldus.psm1'

# Import the Module containing your traffic parser
Import-Module '.\WebDataModule.psm1'

# Set your home and work address
$HomeAddress="Homeroad 1, MyTown"
$WorkAddress="Workaround 1, WorkTown"

# Set a max traveltime limit (in this case, in minutes)
[int] $TravelTimeLimit = 30

# I want it to be under this value for $NumberOfTimes consecutive times
[int] $NumberOfTimes = 3

# Make sure it does'nt get $True on first run
[int] $CurrentTravelTime = $TravelTimeLimit+1

# Reset variable to zero
[int] $NumberOfTimesVerifiedOK = 0

# Run until the traveltime limit has been passed enough times
while ($NumberOfTimesVerifiedOK -lt $NumberOfTimes) {
    # Reset variable
    $CurrentTravelTime = $null

    # Load new data, the "Get-Traffic"-cmdlet is my traffic parser
    [int] $CurrentTravelTime = Get-Traffic -FromAddress $HomeAddress -ToAddress $WorkAddress | select -ExpandProperty TravelTime

    # Check if it is below your traveltime limit, and that it is not $null (cmdlet failed)
    # Increase $NumberOfTimesVerifiedOK if it was ok, or reset to zero if it wasn't
    if ($CurrentTravelTime -ne $null -AND $CurrentTravelTime -lt $TravelTimeLimit) {
    else {
        $NumberOfTimesVerifiedOK = 0

    # Write current status
    Write-Output "Traffic has been verified as OK $NumberOfTimesVerifiedOK consecutive times"

    # Pause for a while before checking again, 10 minutes or so...
    Start-Sleep -Seconds 600

# The while loop will exit when traveltime has been verified enough times.

# Write status
Write-Output "Initiating sunrise, current travel time to $WorkAddress is $CurrentTravelTime minutes, and has been below $TravelTimeLimit for $NumberOfTimes consecutive times."

# Time to initiate the "sunrise effect"

# Set the device id for the lamp you want to light up

# Set start dimlevel
$SunriseDimlevel = 1

# Set how much it should increase everytime we "raise" it
$SunriseSteps = 5

# Set your Telldus credentials
$Username="[email protected]"

# Kick off the "sunrise-loop"
while ($SunriseDimlevel -lt 255) {
    # Write some status
    Write-Output "Setting dimlevel to $SunriseDimlevel"

    # Set the new dimlevel
    Set-TDDimmer -Username $Username -Password $Password -DeviceID $BedroomLampDeviceID -Level $SunriseDimlevel

    # Sleep for a while (30 seconds makes the "sunrise" ~30 minutes long depending on your $SunriseSteps value)
    Start-Sleep -Seconds 30

    # Set the next dimlevel

# Set the lamp to full power (loop has exited) and exit
Set-TDDimmer -Username $Username -Password $Password -DeviceID $BedroomLampDeviceID -Level 255
Write-Output "Maximum level is set."

This script is scheduled to run in the morning (on week days) around the earliest time I want to go up, the first loop will run until traffic calms down, and then start the “sunrise”-loop which will run until the light reaches its maximum level (255).

You could of course turn on other stuff as well, like a coffee brewer (make sure you don’t do this while you are away…), a radio, play some music or something else.

That is one of the (many!) pro’s of doing things with PowerShell! 🙂