Pythonista scripts for EC2

My last post discussed doing web development with an iPad Pro and a VPS, and included Pythonista scripts for automating the creation/deletion of nodes on Linode. Today, I’ll show different versions of these scripts that can be used for starting and stopping EC2 instances.

The scripts assume you have an EBS-backed EC2 instance, which we will start and stop. Unlike Linode and others, EC2 instances of this type do not incur charges when stopped (other than storage), so our scripts will simply start and stop the instance you’ve already created.

To get started, first install boto, a Python library for accessing AWS. The easiest way I’ve found to install it is to run this script, which will install boto in Pythonista, in a folder called boto-module.

Then, in a different folder, save the ec2_start.py script:

Modify it to add EC2 instance ID you want it to start, and the AWS access key you want to use. You’ll be prompted on the first run to enter your secret key, which will be stored in the keychain.

The script will start the instance, and then poll every couple of seconds waiting for the public IP address to be available, which will be copied to the clipboard.

If you want to open another app when the script is completed, pass a URL as a parameter to the script such as workflow://, and that URL will be opened when the script completes. See the prior post for an example of how to do this with the Workflow app.

To stop the instance, use ec2_stop.py:

That’s it – pretty simple, and super convenient for starting and stopping EBS-backed EC2 instances from iOS!

1 Comment

Web development with iPad Pro

As I mentioned last week, I’ve recently been doing a bunch of development and other work with the iPad Pro. This has generally been going very well – the only thing I have to go back to my Mac for, at the moment, is QuickBooks. (And yes, I know there is an online version of QuickBooks – however, there is one thing in it that didn’t work well for me last time I tried it, so for now I’m still using the desktop version.)

Part of what I do on a day-to-day basis is web development, using Ruby and Rails. To do this on the Mac I use Sublime Text, and a couple of terminal windows to accomplish what I need. If we break this down a bit, this is what’s needed for development:

  1. An editor for editing source files
  2. A terminal window of some sort for running specs, deploying code, etc
  3. A browser for testing changes

There isn’t any requirement that that editor needs to be editing files on your local machine, nor that the terminal window needs to be a session on the local machine, nor that the browser needs to point to the local machine. Which forms the basis for a reasonable iPad-based workflow for development.

I’ve been doing this using a combination of a development VPS running Linux, and a few tools on the iPad Pro. The main tool I’ve been using is Coda, which combines an editor, SSH client, and FTP client together in one app. In my experience, Coda is sometimes a bit buggy (random crashes, sometimes freezes for a few seconds at a time, sometimes line numbers go haywire), but for the most part it’s quite polished, and definitely worth using.

And Coda has one feature that really ties everything together – it can edit remote files (on my development VPS) directly, without having to explicitly transfer them. Don’t worry – I’m not editing live files on a production server; rather, I’m editing files live on a development VPS, and connecting to that dev machine using Coda.

When I want to make some changes to the site, I simply:

  1. Start up a new VPS from an image I’ve created
  2. Connect to that image from Coda
  3. Make code changes with the Coda editor
  4. Test changes with a browser pointing to the VPS
  5. Run specs, commit changes, etc. using Coda’s SSH client

All in all, this works just as well as making code changes from the desktop – the only real difference is using the Coda editor rather than Sublime Text.

Optimizing the VPS workflow for Linode

I could just create the development VPS and leave it running all the time, waiting for me to connect and use it. However, it’s almost as easy to start and stop the VPS as necessary, and we’ll save a few bucks doing it this way. Plus we can have some more fun, with Pythonista, which was recently updated for the iPad Pro!

The steps and scripts below are for Linode (shameless referral link), but you can do this anywhere you like. Here are similar scripts for AWS/EC2. But for Linode, this is what we need to do:

  1. Create a development server VPS. Install whatever software you need (e.g. Ruby, etc.), clone your source code into a folder, and basically get everything up and running.
  2. Create a saved image of that VPS, using the Linode Manager tools.
  3. Assuming you have the image saved, you can now delete the original VPS.

You now have a saved image, that you can create a new VPS from any time you like. But that will be a lot of tapping around on the Linode Manager to do so…so instead, we can use a script (which uses the Linode API) to automate this, and run this script in Pythonista.

It turns out this script has to do quite a few things; creating a new node from an image in the Linode Manager hides some of these steps from you (like setting up a configuration profile), but we must do them all manually when using the API.

To get set up, first copy the api.py file from the linode-python GitHub repository to Pythonista. You can copy down the entire repo if you want, but the api.py is the part we need. Then, you can use this script in Pythonista to create a new node:

This script assumes it’s in the same directory as api.py, and further assumes the following:

  1. You have an manually-created image saved, and that image is called “dev image”
  2. You want the new node to be called “dev01”, and be saved in a group called “Dev”
  3. You want your new node to be in the Dallas data center, and you want it to be on the Linode 1024 plan; it will be set up with a 22GB main volume, and a 256MB swap volume, with the latest 64-bit kernel.

You can change any of the above assumptions by editing the code on lines 20-26.

The script will create a new VPS, and will copy the public IP of the new node to the clipboard. From there, you can open Coda, create or edit a site configuration there, and paste in the new IP. In my experience, by the time you open Coda and paste in the new IPs, the VPS will be booted and ready to connect.

To make things a bit more automated, I run the script from a workflow in the Workflow app; here is a screenshot of the workflow I use:

image

It’s quite simple – it just runs the Pythonista script, sending “workflow://” as the return URL, and then opens Coda, with the new IP address on the clipboard ready to paste into your site configuration. I add a shortcut to the workflow to my home screen, and I can spin up a new node with literally one tap. But if you want to, you could do the equivalent without using Workflow at all in this case.

We can also delete the node when we’re done, using another script; be careful with this one, and make sure you completely understand what it’s doing before you run it – if it finds a node called dev01 in the Dev group, it will delete it without confirmation:

Using these scripts and the apps I’ve mentioned, I’ve found development on the iPad Pro (with a keyboard) can be quite similar to the equivalent on a desktop.

11 Comments

Using Working Copy with 1Writer on iPad Pro

Over the last month or so, I’ve been doing development and other work with a new iPad Pro, and the smart keyboard. I sometimes get incredulous looks from people when I tell them about this, so I thought I’d write about some of the things I’ve been doing. First up, editing markdown files in a Github repository.

Working Copy (app store) is one of those apps that seems to fly below the radar – but it’s by far the best git client I’ve found for iOS. It’s free to try, with a in-app purchase to unlock certain features like the ability to push changes to a remote repository. But my favorite part is the way it implements a document provider, allowing you to edit files in your repo with other applications. So here, I’ll show you how to edit markdown files in your repo using another great app, 1Writer (app store).

First, you need to clone your repository into Working Copy. Once you have it down, it will look something like this:

img_0021

Here, this repository is a set of API documentation that I want to edit. Now, we could edit this right here within Working Copy, but instead we will use 1Writer. Open 1Writer, and click the “+” button in the lower right corner:

The first time you do this, you’ll notice Working Copy isn’t in the list. Click “More”, and you’ll be able to enable it:

After you get it enabled, you’ll see it in the list:

Now tap Working Copy, and you’ll see a list of documents you can open from the repository:

img_0025

Pick the document you want (we’ll edit the README.md as an example), and make your edits.

When you’re done editing, go back to Working Copy:

img_0026

You’ll see on the left that README.md is modified, and if you tap on it, you can see the changes that were made. When you’re ready, you can click “Commit” to commit the changes you’ve made, and then push to remote repository if you need to.

2 Comments

Zero to PostgreSQL streaming replication in 10 mins

I’ve found a number of articles on the web for setting up streaming replication in PostgreSQL, but none of them seemed to actually put everything together that I needed, and none of them seemed to use the new pg_basebackup in PostgreSQL 9.1 and later. So with that in mind, here are a set of steps you can use to set up streaming replication, over the internet if you wish, using an encrypted SSL connection. We’re not going to set up log archiving – we’re going to rely solely on the streaming replication for now.

I’m assuming you have a master server set up on Ubuntu 10.04 or 12.04, running PostgreSQL 9.2.x, and you have a new slave server set up on the same OS and pg version. The IP of the master is 1.2.3.4, and the IP of the slave is 5.6.7.8.

First, create the replication user on the master:

Note that we are using REPLICATION permissions, rather than creating a superuser.

Next, configure the master for streaming replication. Edit postgresql.conf (on Ubuntu, this is at /etc/postgresql/9.2/main/postgresql.conf):

We’re configuring 8 WAL segments here; each is 16MB. If you expect your database to have more than 128MB of changes in the time it will take to make a copy of it across the network to your slave, or in the time you expect your slave to be down for maintenance or something, then consider increasing those values.

Then edit the access control on the master to allow the connection from the slave; in pg_hba.conf (/etc/postgresql/9.2/main/pg_hba.conf on Ubuntu):

In this case, 5.6.7.8 is the IP address of the slave that will be connecting for replication, and hostssl means this host can only connect via SSL.

You’ll need to restart the master after making the above changes.

Now on to the slave. In the slave’s postgresql.conf, add the following:

Then restart the slave. No changes are required in the slave’s pg_hba.conf specifically to support replication. You’ll still need to make whatever entries you need in order to connect to it from your application and run queries, if you wish.

That’s all the initial setup we need to do. After you’ve done the above configuration, running the following script on the slave will copy the database over and begin replication (1.2.3.4 is the IP of the master):

That script will stop the slave, delete the old slave cluster directory, run pg_basebackup connecting to the master to copy over the databases (you’ll see the progress as it goes), create a new recovery.conf file, and start the slave. If you look at the logs after this script completes, you should hopefully see messages about it having reached a consistent recovery state.

Be careful – that script is going to delete the old database cluster on your slave, so make sure to read through it and understand what it’s doing.

At this point, you can try writing data to the master database, and you should see it mirrored over to the slave. To check the replication status, you can run the following on the master:

If you found this guide handy, you might also find the PostgreSQL 9 High Availability Cookbook useful as well!

79 Comments
Link

No one is “forced” into piracy

[…] if piracy is actually wrong, it doesn’t get less wrong simply because you can’t have the product exactly when and where you want it at a price you wish to pay.

This. Just because you can’t find the content you want in the form you’d like to have it, doesn’t mean it’s ok to steal it. [via Marco]

Link

Letterpress

A new word game from Loren Brichter (of Tweetie fame). If you love casual games, and you’re tired of Words with Friends, Letterpress is a lot of fun!

MacBook Pro retina pricing

I usually like reading ReadWriteWeb, but I think this article this morning is just link bait (which obviously worked, because I’m linking to it):

Buying a MacBook Pro with Retina means shelling out at least $2,199 for a notebook with a 15.4-inch, 2880×1800 display. Top-end models approach $3,500!

By comparison, the cheapest 15-inch MacBook Pro starts at $1,799, with half the resolution and a different but roughly comparable set of features and specifications. (The Retina version is smaller and lighter but lacks a DVD drive, and uses expensive Flash storage instead of a slower conventional hard drive.)

Anyone who has used an SSD knows there’s nothing “roughly comparable” between the these two computers’ configurations. But we can help them with the math.

Let’s take the cheapest 15-inch at $1799. To get “roughly comparable”, we’ll need to upgrade the RAM to 8GB ($100), and upgrade to the same 256GB SSD ($500), for a total of $2399. You could add another $100 to upgrade to the 1680×1050 hi-res screen, but let’s assume you don’t want that.

To be fair, that machine will have a DVD drive and an ethernet port. So, let’s add those to the base retina MacBook Pro. Base model is $2199, add the USB SuperDrive ($79), and add the Thunderbolt ethernet adapter ($29), for a total of $2307.

So the retina MBP is actually cheaper – AND it comes with 1GB of video RAM, vs. 512MB in the classic MBP.

Is the retina MBP expensive for a laptop? Yes. But none of the 15-inch MacBook Pros are cheap. The new model with the retina display is actually quite aggressively priced, IMHO, as compared with the prior models.

1 Comment

On the MacBook Pro with Retina display

There has been much written about the new MacBook Pro with Retina display. I’ve had one for about a week; I’m not going to write a review, as I’m not sure how anyone could compete with this review…but rather I’ll just mention a few things I’ve noticed in using it for my work.

First, the retina display is quite striking when you use it with applications that have been updated with retina graphics. Most websites that have not updated their graphics don’t look good at all, as I said earlier:

The “blurry” effect is actually more noticeable on the MacBook Pro than it is on the iPad. So for the many, many folks I’ve heard saying “my site looks ok on the iPad, so I’m not going to worry about it” – my recommendation is take a look at it on the MacBook Pro, and make sure you’re comfortable with how it looks. Hint: it probably looks worse than you think.

There are two things that I’ve found a little painful at the moment when using the new MacBook Pro, as Joshua Johnson also noticed.

First, trying to edit 1x artwork on the retina screen is definitely a challenge. At the moment, you just can’t really tell how it’s going to look on a non-retina screen. Maybe a future update to Photoshop CS6 or Pixelmator or some other app will fix this; we’ll have to see.

And second, taking a screenshot on the retina MBP results in an image that’s twice the size you expect; for example, if you’re running at the “ideal” 1440×900 effective resolution, screen shots will be at 2880×1800. That’s great if you need a 2x screenshot to display on a retina display…but if you need a screenshot to display on a 1x display, you don’t have great options. You can downsize it in Photoshop or other editor, but you lose quality.

My solution to both of these problems has been to connect a regular non-retina screen for those tasks. This also has the advantage of letting me see quickly how things are looking on both retina and non-retina screens side by side, at the cost of being tethered to my desk…

All in all, the machine is beautiful, the screen is stunning when viewing high resolution content, and the machine is quite fast as compared to my other machines. I think the issues I mentioned above will probably (hopefully!) work themselves out as the software catches up with the display.

1 Comment

Glassboard 2.0 and Glassbot

The folks at Sepia Labs have released Glassboard 2.0 this morning. Lots of new features in the iPhone and Android apps…but my favorite part is the new web client.

When I first started using the web app a couple of weeks ago, it struck me as a game-changer for Glassboard. Where I would use the mobile apps for casual messages, with the web app I could keep it up on my desktop screen, and send messages and updates much more quickly. I found I was using it for real work.

Which got me to thinking, I’d like a way to automatically post content into a board. For example, maybe I’d like commit notices from Github to be automatically added to a board. Or maybe new support tickets. Or maybe just a way to have some fun with my friends – who doesn’t like seeing a coworker with a huge mustache?

So I created glassbot – a bot for Glassboard…and glassbot_recv, which allows you to post external content into a board.

If you want to try it, you can either:

  • Join the Glassbot Playground board with invitation code cfoli – once you’re in, post a message like “@glassbot help” (without the quotes) to see what he can do.
  • Invite the Glassbot to your own board! Just invite “glassyglassbot@gmail.com” to your board; he’ll accept within a few seconds, and then enter “@glassbot help” (again without the quotes) to see what he can do.

After you invite the glassbot, you can add a post-receive webhook from Github if you want to see notifications – use the URL

http://glassbot-recv.herokuapp.com/github/{BoardId}

as the endpoint URL, replacing {BoardId} with the ID of your board (you can see this in the Glassboard web app).

If you have ideas of things Glassbot could do, or you want to run your own, the code is available at glassbot (in Ruby) and glassbot-recv (Node.js) – add some new actions and let’s have some fun!

Because every board needs a bot.

UPDATE 06/12/2012 – the source code links have been removed at the request of the Glassboard team, while they are making changes to their API. The glassbot is still running, though, and you can still invite it to your boards. I’ll put the code back up when I get the ok.

4 Comments