Amazon has also changed the service’s pricing to encourage people to try the service and stick with it. Moderate usage of the service will be free for at least six months for up to 1GB of data, while the ongoing data storage costs beyond 1GB have been slashed from $1.50 to $0.25 per GB/month.

Finally, a new DomainMetadata API operation will be made available for retrieving statistics about a domain (”database” in SimpleDB parlance) such as the total number of items, and the storage consumed by all of your attribute names and values. Unfortunately the link to the DomainMetadata API documentation link is broken as I write this, but full details are available in the PDF version of the Developer Guide. The DomainMetadata API documentation is now available in both HTML and PDF versions of the Developer Guide.

CloudFront uses two techniques to deliver your data quickly. It caches S3 objects in the network’s edge locations (for a configurable amount of time), and automatically routes incoming requests to the edge location nearest to the requester. If you provide content to people who are distant from an S3 server location, this new service could significantly improve download speeds for your users.

The price for distribution is based on the amount of data transferred and the number of requests, with differential pricing depending on the edge location that serves the content. Bear in mind that you will also incur the standard S3 service costs whenever an object is provided to an edge location.

It is straight-forward to create a CloudFront “distribution” based on one of your S3 buckets. The AWS blog announcement mentions a number of tools and libraries that include support for the new service. My own JetS3t Java library has also been updated with basic API support for the new service, though for now you will need to download the latest code from CVS to access this feature. If you are keen to get your hands dirty, go get a CVS checkout and review the sample code in org.jets3t.samples.CloudFrontSamples.

Inevitably, there are some drawbacks to the service:

• Only publicly readable objects can be distributed through CloudFront, so you cannot use access control settings or temporary signed URLs to limit content delivery to specific people.
• You cannot manually (or programmatically) expire objects at an edge location once they have been cached. Be careful to use appropriate caching directives if you intend to update or remove distributed objects.
• You cannot set a cache expiry time less than 24 hours, so the service is not appropriate for data that is frequently updated.
• As with S3, there is no way to display a default index.html page when someone requests the root location of your CloudFront distribution. This makes it difficult to use the service as a complete replacement for a web server.
• There is currently no support for request logging, though account and usage reports are available.
• There is not yet an edge location close to my Australian home

Despite these drawbacks, CloudFront is a very promising new tool for anyone who needs to distribute a lot of content to a broad audience.

The service has graduated from “beta” to production level, which means that a Service Level Agreement is now available. If you experience uptime below 99.95% in a given month and you can provide supporting evidence, Amazon will provide a 10% service credit for that month’s bill.

Even more interesting is the news that you can now run Windows Server 2003 and SQL Server on EC2 instances. Amazon’s Windows on EC2 documentation lists the prices for running these instances: from 12.5¢ per hour for a small instance with Windows only, and from $1.35 per hour for an instance with Windows, SQL Server, and Authentication Services. See the Windows AMIs listing for more details.

Understandably, Windows instances on EC2 work very differently than the Linux ones. The folks at RightScale have published a blog post describing some of the differences and quirks of the Windows instances.

Although I much prefer “Unixy” platforms for my own development, I can imagine situations where it would be very handy to have a Windows machine easily available — such as for running those vital but irritating programs that are only made available for Windows. Australian Tax Office, I’m looking at you…

To take advantage of the new Windows instances, download the latest version of the ElasticFox Firefox Extension and be sure to check out the Elasticfox Manual (PDF).

Here are the steps I took to clean up the database and data directory — with extreme prejudice.

In the psql console:

VACUUM FULL VERBOSE;
REINDEX DATABASE mydatabase;

On the command line:

pg_resetxlog /path/to/my/datadir

Running all of these steps is a brutal process. Do not run the pg_resetxlog command if you have important data, because you will probably want to keep your write ahead log.

In this post, I will show how to extend the SimpleDB ruby client implementation from my book Programming Amazon Web Services (O’Reilly 2008) to support this new operation, and give some usage examples.

Let’s start by adding a new method query_with_attributes to the SimpleDB Ruby sample code (available from http://examples.oreilly.com/9780596515812/). Edit the file ruby/SimpleDB.rb and add the following method definition after the existing query method. If the code below looks wonky in your browser, you can download a file instead

def query_with_attributes(domain_name, query_expression=nil,
                          attribute_names=[], options={:fetch_all=>true})
  more_items = true
  next_token = nil
  items = []

  while more_items
    parameters = build_query_params(API_VERSION, SIGNATURE_VERSION,
      {
      'Action' => 'QueryWithAttributes',
      'DomainName' => domain_name,
      'QueryExpression' => query_expression,
      'MaxNumberOfItems' => options[:max_items],
      'NextToken' => next_token
      },{
      'AttributeName' => attribute_names,
      })

    xml_doc = do_sdb_query(parameters)

    xml_doc.elements.each('//Item') do |item_node|
      item = {'name' => item_node.elements['Name'].text}

      attributes = {}
      item_node.elements.each('Attribute') do |attribute_node|
        attr_name = attribute_node.elements['Name'].text
        value = attribute_node.elements['Value'].text

        if respond_to? :decode_attribute_value
          # Automatically decode attribute values if the method
          # decode_attribute_value is available in this class
          value = decode_attribute_value(value)
        end

        # An empty attribute value is an empty string, not nil.
        value = '' if value.nil?

        if attributes.has_key?(attr_name)
          attributes[attr_name] << value
        else
          attributes[attr_name] = [value]
        end
      end

      item['attributes'] = attributes
      items << item
    end

    if xml_doc.elements['//NextToken']
      next_token = xml_doc.elements['//NextToken'].text.gsub("\n","")
      more_items = options[:fetch_all]
    else
      more_items = false
    end
  end

  return items
end

The QueryWithAttributes operation works almost exactly like the SimpleDB service’s Query operation, except it includes item attributes in the result. You can optionally specify exactly which attributes to retrieve using the attributes parameter, or leave this empty to retrieve all item attributes.

Continuing on from the stock quote database example in my book, here is the command you would issue to retrieve the records for days on which more than 70 million Apple stocks were traded. Notice that I am taking advantage of the new sorting capabilities of SimpleDB to retrieve ordered results:

irb> query = "['Volume' > '#{sdb.encode_integer(70000000)}'] sort 'Volume'"
irb> sdb = SimpleDB.new
irb> sdb.query_with_attributes('stocks', query)
=> [{"name"=>"AAPL-2007-07-26T00:00:00Z",
  "attributes"=>

   {"Code"=>["AAPL"],
    "High"=>[148.5],
    "Open"=>[145.91],
    "Close"=>[146.0],
    "Date"=>[Thu Jul 26 00:00:00 UTC 2007],
    "Volume"=>[78093900],
    "Low"=>[136.96],
    "Adjusted Close"=>[146.0]}},
    . . .

If you were only interested in the Open and Close attribute values, you could tell the service to include only these attributes in the results:

irb> sdb.query_with_attributes('stocks', query, attributes=['Open','Close'])

It is a good idea to retrieve only the attributes you really need, because the SimpleDB service limits each response message to 1 MB and you will get more results in fewer requests if you reduce the amount of data you are asking for.

The QueryWithAttributes operation makes it much easier to use SimpleDB because you no longer need to perform follow-up queries to retrieve attribute data. However, there may be a price to pay for this convenience so be sure to run some tests before converting your applications. It may turn out that, in some circumstances, you can retrieve results more quickly using the Query/GetAttributes approach with multiple request threads on your client, than you can using the QueryWithAttributes operation.

In early forum discussion it looks like the new operation is both the easiest and fastest way to perform queries, but results may vary depending on how you use the service, your dataset, and your network performance.

If you are using SimpleDB with a large data set or complex queries, try the new operation and discuss your experiences on the forum so we can gain a better idea of the strengths and weaknesses (if any) of QueryWithAttributes.

EBS provides for block level storage space that can be attached to your EC2 instances, but which exists separately from these instances and persists even if an instance fails or is shut down. The storage volumes can be up to 1 TB in size, can be assigned to any one of your instances on-demand, and can be easily backed up as a snapshot to S3.

The EBS mechanism will make it significantly easier to store data reliably on EC2 instances, removing one of the main pain points of using EC2 instead of a standard data centre.

I don’t yet have much experience with EBS myself but there is a wealth of information available for EC2 users in the posts listed below, and third-party tools and libraries like ElasticFox, Typica and boto have been updated to support the storage volumes.

• The announcement: Amazon EBS (Elastic Block Store) - Bring Us Your Data
• A growing list of tools and libraries that support EBS: Amazon EBS - Tool and Library Support
• Typically excellent, in-depth discussion of the EBS feature from the folks at RightScale: Amazon’s Elastic Block Store explained and Why Amazon’s Elastic Block Store Matters

The new fee structure makes the FPS micropayment instruments much more attractive. Using the prepaid and postpaid instruments, you can aggregate many small payments into a few large transactions to minimise the cost of transaction fees over these payments. Previously, Amazon took a cut of each individual payment as well as imposing transaction fees to fund and settle the instruments, but that extra fee burden has been removed.

This release includes bug fixes, application improvements, and support for some of the newest features of Amazon’s Simple Storage Service (S3). Highlights include:

• Support for copying, moving, and renaming S3 objects with the programming toolkit and Cockpit application
• The metadata associated with objects can now be updated without the need to upload an object again
• Improved data verification options for ensuring that information is not corrupted in transit
• The Synchronize application supports new options to improve performance and decrease memory requirements when synchronizing large numbers of files.

You can download version 0.6.1 from the JetS3t Downloads page. The web site has also been updated with the latest application manuals, code samples, and API documentation.

Enjoy!

Sorting still suffers from some limitations: you can only sort by a single attribute, you cannot perform sorting in queries that contain union or not predicates, and the attribute against which you are sorting must be present in a query predicate. However, it is still a massive improvement on sorting query results in your own client code.

For example, given the following SimpleDB query that finds historic stock records for a given time period:
['Code' = 'AAPL'] intersection ['Date' > '2007-12-01']

With the new sort operator, you can now instruct the service to return the results in reverse (descending) date order like so:
['Code' = 'AAPL'] intersection ['Date' > '2007-12-01'] sort ‘Date’ desc

Although you can only sort by an attribute that is also mentioned in a predicate, you can easily work around this restriction by including a starts-with '' predicate for that criteria, which will always evaluate to true. Here is a query to return stock records sorted according to the closing price, from lowest to highest:
['Code' = 'AAPL'] intersection ['Close' starts-with ''] sort ‘Close’ asc

The SimpleDB update also includes some other new features, such as a new does-not-start-with operator and support for 10 predicates per query instead of 5, but I think it is the sorting that will have the most impact.

Update

Amazon has posted a full explanation of the recent S3 outage here: Amazon S3 Availability Event: July 20, 2008.

This makes for interesting reading, but ultimately the lesson from this most recent outage is that service failures can and will happen. Even to Amazon.

If you are building a cloud-based application you should try to include as many contingency options as much as possible, but few contingencies could have survived that outage. Let’s hope it isn’t repeated.