Planet Eucalyptus

After having heard about FOSDEM for long time, and this year I had the privilege to attend it. This may be the reason of the extreme cold  spell that hit Europe at that time. I reached Brian Thomason in Hamburg where he presented Eucalyptus. They loved him, and I have to agree with them: the presentation was very interactive and Brian was very charming. We then reached Brussels by train, on the only non-heated car: needless to say I was ready for a hot bath once we reached the hotel.

FOSDEM was as good as I read about, and more. My interests brought me most of the time at the Cloud and Virtualization track, although I did walked into a Google Summer of Code talk, CentOS and Debian talks, and a survey on how to have HA with MySQL. I like the HA talk (search for Ivan Zoratti's HA reloaded): there is no free lunch when deploying HA, and the talk highlights how decisions need to be taken early on about what kind of  HA one wants and she is willing to pay for.

The Cloud and Virtualization track was very well attended. It was good to see old friends and shake hands, from Xen's Lars, to Dave and James from the Ubuntu server team, Thierry and Rick and more. Very good content too. I did enjoy quite a bit the libguestfs talk, since lately I have been doing some images work,  and I already have been using zerofree, but I learned that there are a lot more tools I can use to simplify my life.

Update 2/20/2012 – after further discussion with EuCa folks, I’ve updated and expanded the responses below. Thanks to everyone who chimed in!

As the Eucalyptus community picks up steam, more and more folks who aren’t Eucalyptus employees are giving talks about EuCa at various events around the world – which is fantastic. How do we get everyone the information they need to give a successful talk? Ah, that’s the fun challenge.

Sometimes we just fill in gaps as we go along. For instance, Shaon’s talk, “Next generation cloud deployment: Self help is the best help!” just got accepted to SoftExpo 2012. Subsequently, a few of us had a spontaneous discussion on how to field a couple tricky questions when giving EuCa talks – the post below is based on these meeting logs. (Disclaimer: these represent individual opinions and don’t speak for Eucalyptus as a company, etc.)

Why is Eucalyptus using the Amazon API?

Because AWS is way farther ahead than the others, and we believe it to be the most worthy of focus. We may support other APIs in the future, but we prefer to focus now on the best API, and that’s AWS. See this post by Mark Shuttleworth for reference – it’s about OpenStack, but makes the same point.

Comments from Eucalpytus folks: “AWS is by far the most popular public cloud out there. I.e. it’s not just best, it is also the most common.

How much does Eucalyptus support cost?

Varies widely. Contact Eucalyptus for details.

I originally posted this without a number because I know companies are often hesitant to post that sort of pricing information on the web. After asking around within Eucalyptus, I found out they actually wanted to give a public answer – awesome! Therefore: if we change the question to “subscription” (not “support”), the answer is “It starts at $2,000 per server per year.”

What third party apps are available for Eucalyptus management?

There are a few options for apps that run on Eucalyptus and provide Amazon-RDS-like support. Three popular ones:

• HybridFox
• RightScale
• Enstratus

Providers of IaaS services need a usage monitoring tool. What does Eucalyptus offer?

There are hooks in Euca 3.0 for reporting usage; this is a new feature. We don’t have [public] details yet, but they will be in the docs when we release them (which will hopefully be soon).

In Euca 2.0, instances cannot be recovered once the node goes down. Is there any solution?

Boot from EBS is a new feature in Euca 3, and may be able to help with that problem somewhat. However, redundancy (multiple instances running at all times) is the way to survive instance failures. If you have multiple zones, start instances on all zones.

Instances are supposed to be expendable. HA (High Availability) is for the infrastructure: the app will still need to be designed with the cloud in mind to be fully HA. EBS and Walrus are the persistent storages to be used to keep the states and backup.

Wait, wouldn’t EBS and Walrus use more resources? I thought cloud was supposed to minimize resource usage.

Yep. HA wastes resources by design.

A Eucalyptus employee commented with a clarification here: I wouldn’t say that HA “wastes” resources. It of course uses more resources, but for a good and intended purpose. So it’s not a waste. I would say something like “there is nothing such as a free lunch. To have HA, you need redundancy, and redundancy requires its own resources.”

What are the alternatives to Eucalyptus, and why would someone choose Eucalyptus over them?

It was pointed out to me that the first question here needs to be “how do you define ‘alternatives’?” But in any case, here are a few.

• A suggested addition to the list: vCloud Director from VMware, which is also aimed at enterprise customers – but is not open source.
• Openstack: Broad community supported by many companies, modular design. Both a blessing and a curse. Openstack is much more a set of tools for building a cloud; Euca is cloud-in-a-box. And openstack, honestly, just isn’t as far along. (Additional community comments: it’s designed primarily for public clouds and doesn’t support the AWS API.)
• Cloudstack: Good product, good UI. Integration with AWS isn’t as good. (Additional community comments: it’s also mostly used by service providers, not enterprises.)
• Opennebula: Again, more of a toolkit approach. Not too many components — just more flexibility about how you put them together. Seems like Openstack and Opennebula are both good for the service provider market, and Euca and cloudstack are more all-inclusive products for the enterprise market.

That’s all we had time for — what other questions would you ask, and how would you answer these?

I’ve gotten some feedback on my post on fielding common questions at your Eucalyptus talk since it came out nearly 2 weeks ago, and have updated the text accordingly — check it out if you’re curious. I was actually urged (by EuCa employees) to put pricing information there – a level of transparency that surprised even me.

I’d also like to give a shout-out to Dr. Karl Wurst, who some of you have seen around the Eucalyptus IRC channels recently. Karl chairs the CS department at Worcester State University and is a long-time member of the Teaching Open Source community who’s been getting his students involved in open source projects since 2010. He’s taking his junior/senior Software Development class into Eucalyptus as their spring term project, and they have taken on the challenge of testing eutester against the new 3.0 release – no small feat, considering that they’re testing new test software against newly-released software with no prior experience with the platform.

I predict the readability of Eucalyptus getting-started documentation will dramatically improve over the coming weeks as they progress – which is incredibly important if we want new folks to pick up on the project. Most people fail silently; if they can’t get something to work, they’ll quietly give up and go away, and you’ll never be the wiser. By committing to fail publicly and loudly, Karl’s class is taking a vital role (and one that requires no small amount of bravery). They speak for the people who won’t. And as newcomers, they’ll be able to write better explanations for other newcomers than all the old-timers out there. Fresh eyes are an asset; if you have them, use them.

His students are blogging as they go along, and it’s interesting to see their take on the project from a newcomer’s perspective. If you see them on IRC or the mailing lists, say hello and introduce them to whatever you’re working on – and if you see something interesting on their blogs, drop by and leave a comment. Those sorts of small contacts with the “real world” are ordinary everyday things to those of us who are used to the open source world (or heck, industry in general), but trust me; they’re absolutely magical the first time you start getting them as a student. (I still remember being awed as an undergrad that people were emailing me about things that weren’t homework.)

So welcome, Worcester State! Welcome to the wild and wooly wide, wide world of Eucalyptus. Glad you’re here.

Leading Private Cloud Platform Provider Taps Industry Veteran Ning Wang

SANTA BARBARA, Calif. – February 16, 2012 – Eucalyptus Systems, creator of the most widely deployed on-premise cloud computing platform, today announced that it has expanded its leadership team with the addition of Ning Wang as its Chief Financial Officer. Co-Founder Woody Rollins, who previously held the CFO role, has been appointed to Chief Talent Officer, responsible for developing the team to capitalize on the growing demand for private cloud architectures.

"Woody's contribution to Eucalyptus since its founding has been immeasurable— he knows what it takes to build a great team and a successful company," said Marten Mickos, Eucalyptus CEO. "In addition to his move, we welcome Ning Wang as our new CFO. Ning has an outstanding financial track record and unique experience managing rapid growth in technology companies. With this addition to our executive team, we are positioned for global leadership in our industry segment."

With nearly 20 years of experience, Ms.

read more

Dasher Technologies to Contribute on a Solution That Enables Enterprises to Validate Hosted Private and AWS-Compatible Hybrid Clouds

February 13, 2012 – CLOUD CONNECT – CoreSite Realty Corporation (NYSE: COR), a U.S. provider of powerful, network-rich data centers, today announced that Eucalyptus Systems, creators of the most widely deployed on-premise cloud computing platform, has launched a new enterprise-class cloud test lab offering at CoreSite’s Bay area data center campus to address the growing demand for cloud computing technologies. The Eucalyptus Cloud Test Lab provides enterprises with a secure, fully functional hosted infrastructure as a service cloud to validate their cloud application migration strategies prior to implementation. The official product launch date is set for March 1, 2012.

Eucalyptus Cloud Test Lab is a complete cloud-ready data center solution with fast and easy setup.

read more

Day 2 started with a real good breakfast. Again thanks to the organizing team for arranging such a good breakfast. I should say I really loved the weather at Pune after staying at Jaipur for past 1 month specially when it is cold. My prime goal for day 2 was to have the lab ready [...]

I got the opportunity to attend as well as speak at gnuNify 2012 , Pune. It was my first FOSS conference as a speaker and it went really great. I was there for two days 10th and 11th Feb and was accompanied by Atul Jha one of our Eucalyptus community member in India. Thanks to [...]

All right, I know the open source world has fast turnaround times, but this is just ridiculous. And awesome.

You may have seen my first attempt at a business card design for Eucalyptus community members.  I sent it out thinking that maybe there would be feedback in a week or so, and I’d crank out a v.2.0 then…

It took four hours.

I woke up this morning to find that Jef van Schendel, a design student in the Netherlands who hacks on Fedora, had taken my design draft and made an svg mockup with color matching, more whitespace, and better alignment. Simultaneously, David Butler, the VP of Marketing at Eucalyptus, dropped a “contributor” logo variant into my inbox.

And so less than 24 hours after the first design was posted, we have an infinitely better one. BAM.

Helpful hints for future Euca swag designers: Eucalyptus-blue is #003f5e or 0, 63, 94 in RGB, and Eucalyptus-green is #8cc63f or 140, 198, 63 in RGB. Also, here’s how to make custom colors in LibreOffice.

The LibreOffice file is available for download here. It’s basically Jef’s design converted to LibreOffice and using David’s logo. As with the first card design, you’ll need the Gillius ADF font – ttf-adf-gillius from Ubuntu repositories, adf-gillius-fonts in Fedora ones, or just download Gillius Collection fonts directly.

Finally, a reminder from Greg:

Because we don’t have our trademark policy yet, use of this design is restricted to those who have explicit permission to do so. Which we will give quite liberally, but still, that permission is legally required to ensure that we maintain legal control over our trademarks.

So if you want that permission, find us on IRC (#eucalyptus, irc.freenode.net) or the mailing list (http://lists.eucalyptus.com/cgi-bin/mailman/listinfo/community) and let us know of your intention to use these cards. If you’re even considering this, it’s likely to be a very brief conversation that ends with us saying “approved, go for it.”

Happy conferencing! And remember, release early, release often. You never know who’s watching!

Update 22 hours later: Jef and Dave sent in card and logo redesigns, and we now have a much-improved second version of the card, which you should use instead. Thanks, guys!

Folks, we’ve got a business card template. Here’s a preview.

Disclaimer: I’m not a graphic designer, as this template makes painfully obvious. I threw this together in 16 minutes (yes, I timed myself) so it’s a very basic design. The colors and fonts don’t even match the Eucalyptus logo. However, you can use this design to run a pre-cut business card sheet through a home or hotel printer, and that’s what we need, because people are going to events tomorrow. (Actually, at this hour, I think it may even be today already.)

The template is a LibreOffice file, so you’ll need that installed before you can edit it. (LibreOffice is cross-platform and commonly available in Linux distributions – OpenOffice would work too.)

You’ll also need Gillius ADF, a Libre font from the “Gillius Collection” – download it here. I chose this font because it’s also commonly available in the repositories of Linux distributions. Ubuntu calls it ttf-adf-gillius and Fedora calls it adf-gillius-fonts so you can yum or apt-get install to your heart’s content.

Once you have those prerequisites installed, head to this ticket to download the template. Enjoy – and if you know graphic design better than I do, please feel free to fix.

Being a system administrator is the easiest job in the building when the system is working; no one questions your presence nor existence. Your tasks are highly under-appreciated during the time of peace, yet you do not mind for such vanity since you’d rather be reading blogs and watching youTubes in serenity. Every once in a while, an idiot cracks an ancient joke, “hey, aren’t you supposed be working?”. But, I’m working, you imbecile employ-of-the-month.

However, the curse begins once you step out of the building, entering the realm of unknowns, far-disconnected from the comfort of your Macbook and Wi-Fi. While staring at endless tail-lights on a freeway, having a long walk on the beach, or being queued behind 7 shopping carts at a grocery market, your mind begins to wonder, “how’s my system doing?”

Since the day 1, you have set up numerous layers of email-notification alarms, but it’s never enough; “getting the e-mails” only means “it’s too late”. Always there is an urge of logging in. But you can’t. You are cut off. You are trapped; the lady in front of you just pulled out a checkbook while the sign clearly says “credit or cash only.” You begin to panic. You compulsively refresh emails on your smartphone, but no answers. Silence is deafening. No news is never the good news. The only exit is when the system whispers in your ear: ”Have no worries. Everything is working… For now.”

Now, your concerning days are over. In the midst of the 3G wilderness, the application Pigeons on a Euca will deliver you peace and tranquilly that are comparable to those of a laptop on VPN. Of course, that is if you are equipped with a smartphone at all times and the system is running Eucalyptus Cloud.

The trick is to run a periodic Cloud monitoring app via Twitter.

Instead of being passively notified by emails when there arise problems in the system, you can set up the application Pigeons on a Euca that runs a small script that actively “tweets” the status of the cloud for you and your co-sys.admins to follow.

Screenshot of Pigeons on a Euca on iPhone

Here are the requirements:

• Have a twitter account opened for this application.
• Have a machine, or a virtual machine, running Linux with network capability.
• Have the cloud admin’s credentials.
• Have a smartphone with Twitter Client App installed.

Current (Beta) Features**:

• In every 1 min, it tweets the status-change on running instances*.
• In every 10 min, it tweets the number of currently running instances in the cloud*.
• In every 10 min, it tweets the number of newly-launched instances in the cloud*.
• In every 10 min, it tweets each availability zone information

* These features rely on the new version of euca2ools (v 2.0)

** The application is highly configurable so that more reporting can easily be added when needed.

Instructions on How to Set up the Application Pigeons on a Euca

First, you need to open a new twitter account.

1. Go to twitter.com and sign up for a new account; if you already have an account with Twitter, you are going to need a new email address to create a new twitter account for this application.

2. As soon as the new twitter account is open, check the “Protect my Tweets” box on the Tweet Privacy section so that your tweets do not accidently get broadcast to the public channel.

3. Apply for a developer account at dev.twitter.com.

4. After opening the twitter developer account, you need to create an application at dev.twitter.com.

5. Fill out the application details form. There is no requirement on what you put in the name and description boxes. For the website box, you may specify any working web URL of your choice; it won’t matter for this application.

6. After creating the application, on the “settings” page, change the access level to be “Read and Write“

7. Click the button “Change this Twitter application’s settings” at the bottom of the page to apply the change. It might take a few minutes for the change to be applied.

8. On the “Details” page, verify that the access level is changed to “Read and write“. After seeing that the change has taken place, click on the button “Create my access token” at the bottom of the page.

9. Go to the page “OAuth tool” and verify the consumer key and access token are generated. You will need these keys to configure the application Pigeons on a Euca later.

10. At this point, your twitter account is configured to receive script-generated tweets from the application Pigeons on a Euca.

Second, after the twitter account is ready, you need to set up the machine where the application Pigeons on a Euca will be running on.

1. Install a perl module “Net::Twitter::Lite” on your Linux box.

You may install the module from source by visting the link:

http://search.cpan.org/~mmims/Net-Twitter-Lite-0.10004/lib/Net/Twitter/Lite.pm

Or, for UBUNTU distributions, such as Lucid, you may simply add the line:

deb http://ubuntu.mirror.cambrium.nl/ubuntu/ lucid main universe

to “/etc/apt/source.list”.

Then, install the perl module twitter-lite-perl by using the commands:

apt-get update
apt-get install libnet-twitter-lite-perl

2. Install the latest version of “euca2ools” (v 2.0)

Visit the website (http://open.eucalyptus.com/downloads) for detailed instructions on how to install the latest euca2ools.

For UBUNTU distributions, such as Lucid, you add the line:

deb http://downloads.eucalyptus.com/software/euca2ools/2.0/ubuntu lucid universe

to “/etc/apt/source.list”

Then, install euca2ools by using the commands:

apt-get update
apt-get install euca2ools

Third, after all the necessary modules and tools are installed on the machine, now you can finish setting up the application on the machine.

1. Download the tarball pigeons_on_a_euca.tar.gz from the project repository:

https://projects.eucalyptus.com/redmine/projects/pigeons-on-a-euca/files

or

https://github.com/eucalyptus/pigeons_on_a_euca

2. Untar the tarball at a directory of your choice:

tar zxvf pigeons_on_a_euca.tar.gz

3. On the “my applications” page on dev.twitter.com, copy the lines in the “OAuth tool” section, as shown in the Step 9. of the first instruction set.

4. Change the lines in the file “./pigeon_on_a_euca/pigeon_cage/key/o_auth_settings.key” with your account’s actual values.

5. Perform a quick check to validate the setups so far by running the commands:

cd ./pigeon_on_a_euca/pigeon_cage

perl ./tweet_it_away.pl ./tweets/mytest.tweet

6. Check the twitter account to verify that the line “this is a test” was tweeted. Also notice the lock sign on the tweet that indicates the security level is private.

7. After verifying the test tweet, go to the directory “./pigeons_on_a_euca/credentials” and store your Eucalyptus cloud’s admin credentials.

8. Verify that you can talk to your Eucalyptus cloud via the admin credentials by running the commands:

cd ./pigeons_on_a_euca/credentials

source eucarc

euca-describe-availability-zones verbose

9. At this point, the application is all set to run. Do a quick check by running the main script:

perl ./activate_the_pigeons.pl

10. Check the twitter account to verify that the status of instances running on the cloud are being tweeted.

11. To run the main script in the background, do:

nohup perl ./activate_the_pigeons.pl > ./stdout.log >> ./stderr.log &

12. To monitor the run:

tail -f stdout.log

Last, install any Twitter Client App on your smartphone and follow the account that you created above.

Now you have an mobile application that keeps you updated with the status of the cloud.

Warning: The amount of tweets generated by the application might be overwhelming; at its maximum rate, it will upload 350 tweets per hour. It is recommended that you and your co-workers open a separate twitter account exclusively for receiving tweets from this application.

And, if you decide to modify the script, please be aware of the hourly limit of the tweet updates, which is set to be 350 tweets per hour. Carefully limit your tweets so that the application maintains consistent tweet-ability.

Thank you for your interest in the application, and feel free to contribute and share.

Kyo

A few weeks ago, I posted a blog entitled “Fun with Varnish and Walrus on Eucalyptus, Part 1“. This blog will follow-up on my blog to showcase a few production use cases that utilize the Varnish-Walrus architecture built on top of Eucalyptus.*NOTE* This architecture can also be leveraged using AWS EC2 and S3. This is [...]

I've always been a fan of the flexibility of anaconda and kickstart not just for installing systems, but also for rescuing a system when something goes horribly wrong.  Yesterday I updated a remote test system from Fedora 16 to Rawhide, and I found myself with no network access to the machine due to a firmware issue.  The system has DRAC 6 express, so I can reset the system and force a pxe boot, but I can't see or interact with the console when it boots.   Recent Fedora releases have a great way to rescue a system in this state.  First, you set up a kickstart file for the rescue (probably only the first two lines are needed, but I did not test with fewer lines than this):

rescue --nomount
sshpw --username=root sekrit --plaintext
url --url http://mirror.eucalyptus/fedora/releases/16/Fedora/x86_64/os/
lang en_US.UTF-8
firewall --enabled --port=22:tcp

Then set up these boot options in your PXE configuration:

ks=http://yourWebServer/ks/fedora-16-rescue.cfg ksdevice=link keymap=us lang=en_US sshd

This works just like rescue mode always has, except you don't need console access. Very cool.

I'm sure this feature isn't news to a lot of Fedora users, but sometimes cool new features like this sneak into a Fedora release and not everyone realizes it, so it seemed to be worth a quick blog.

My last blog post was a long and quite hackish procedure for running a Fedora install on a live instance in a Eucalyptus 3 cloud... and now I'm going to show you the easier way to build an image.  I spent some time kicking around ami-creator, and I only ran into a few small issues.  I've forked it on github and committed the necessary changes.  There is a sample kickstart file in the source tree.  Installation is a snap (sorry for not having it in rpm form, but that wasn't the goal of the day):

• easy_install ez_setup
• git clone https://github.com/eucalyptus/ami-creator
• cd ami-creator
• python setup.py build
• python setup.py install
• mkdir ~/f16-image
• cp ks-fedora-16.cfg ~/f16-image/
• cd ~/f16-image
• optionally, go modify the kickstart file to point to your mirror, add the packages you want, change the disk size, etc.
• ami-creator -c ks-fedora16.cfg -n f16test -v -e

When  the process completes, you'll have a few new files in the current directory:

• f16test.img
• initramfs-3.2.2-1.fc16.x86_64.img
• initrd-plymouth.img
• vmlinuz-3.2.2-1.fc16.x86_64

You can ignore initrd-plymouth.img.  Just go through the normal steps of bundle, upload, and register for each of the other three files, and you should have a working Fedora EMI.  It can't get much simpler than that.  Thanks to Jeremy Katz for starting the ami-creator project.  I hope that someday we'll see this rolled into the live image tools project where it belongs.

This post is the result of a challenge given to me by Seth Vidal, which showed up in his weekend blog post.  He was musing about whether it's possible to actually do a kickstart, or even an interactive install, in a cloud instance. I have to put some disclaimers around this post, because I am _not_ advocating this approach, and I'm going to show you a feature of Eucalyptus 3 that could void your warranty if used in anger. As my friend Michael likes to say, if you break it, you get to keep both pieces.

What we hashed out on Friday was that, in order to be able to kickstart inside an instance, you have to be able to pass boot parameters. In Eucalyptus 2, the only real way to do this was by patching the node controller with something similar to the NEuca patches. In Eucalyptus 3, we've implemented a sort of "escape hatch" called nc-hooks to allow folks to customize behaviors at instance definition and launch time. There's an example shell script in /etc/eucalyptus/nc-hooks/ which shows how you might write your own hooks.

Knowing that the nc-hooks feature existed, I had to think about exactly how to pass boot parameters and get them into libvirt.xml before instance launch. Passing them via userData was the obvious choice.  I came up with a couple of xslt files and this script to make the magic happen:

#!/bin/sh  

event=$1
euca_scripts=/home/eucalyptus/scripts
inst_home=$3

rewrite_libvirt_xml() {
  # Get only the value of the "bootparams=..." line from userData
  BP=$( xsltproc $euca_scripts/get-user-data.xsl $inst_home/instance.xml \
       | base64 -d \
       | sed -r "/bootparams=/!d; s/^.*bootparams=(.*)/\1/" || exit 1 )

  # Substitute the value of $BP into the stylesheet
  sed -e "s!@@BOOTPARAMS@@!$BP!" < $euca_scripts/insert-boot-params.xsl \
       > $inst_home/insert-boot-params.xsl || exit 2
 
  # Rewrite and replace libvirt.xml for this instance
  xsltproc $inst_home/insert-boot-params.xsl $inst_home/libvirt.xml \
       > $inst_home/libvirt.xml.new || exit 3
  cp $inst_home/libvirt.xml $inst_home/libvirt.xml.orig 
  mv -f $inst_home/libvirt.xml.new $inst_home/libvirt.xml 
}

case "$event" in
    euca-nc-pre-boot)
        rewrite_libvirt_xml
        exit 0
        ;;
    *)
        exit 0
        ;;
esac

I don't have a vast amount of experience when it comes to xml processing, so forgive the horror of these stylesheets. The first one, get-usr-data.xsl, is quite simple:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output encoding="UTF-8" indent="yes" method="text"/>
    <xsl:template match="/instance">
        <xsl:value-of select="/instance/userData"/>
    </xsl:template>
</xsl:transform>

The second is a little stranger, and was done with some help from StackOverflow:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output encoding="UTF-8" omit-xml-declaration="yes" indent="yes" method="xml"/>
  <xsl:template match='node()|@*'>
    <xsl:copy>
      <xsl:apply-templates select='node()|@*'/>
    </xsl:copy>
  </xsl:template>
  <xsl:template match="cmdline">
    <cmdline>@@BOOTPARAMS@@</cmdline>
  </xsl:template>
</xsl:transform>

So with these files in place, I now need to configure an installer kernel and ramdisk.  These come from the /fedora/releases/16/Fedora/x86_64/os/images/pxeboot/ directory of your favorite Fedora mirror site.  The kernel and ramdisk registration process is the usual:

• euca-bundle-image --kernel true -i vmlinz
• euca-upload-bundle -b f16 -m /tmp/vmlinuz.manifest.xml
• euca-register f16/vmlinuz.manifest.xml
• euca-bundle-image --ramdisk true -i initrd.img
• euca-upload-bundle -b f16 -m /tmp/initrd.img.manifest.xml
• euca-register f16/initrd.img.manifest.xml

I don't really *need* a disk image here, but an EMI cannot be registered without one, so I fake it:

• dd if=/dev/zero of=fake-emi.img bs=1k count=10000
• mke2fs fake-emi.img
• euca-bundle-image -i fake-emi.img
• euca-upload-bundle -b f16 -m /tmp/fake-emi.img.manifest.xml
• euca-register --kernel eki-EA183EA8 --ramdisk eri-6ED23EF2 f16/fake-emi.img.manifest.xml

Note that even if you aren't trying to do key injection, the disk image needs to have an ext2-compatible filesystem on it.

Next, I need a volume to install into:

• euca-create-volume -s 10 -z PARTI00

Before I boot the instance, here's where I have to be honest with you readers: I make a lot of mistakes when I test things like this.  Typos, logic errors, you name it. So for debugging purposes, I uncomment this line in /etc/eucalyptus/libvirt.xsl:

<graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>

You definitely should not have this line uncommented for normal use, as it will allocate a port for vnc for every instance you launch, and without some extra configuration, it doesn't even require a password to connect. For quick debugging on a safe network, though, it's a good way to see what's going wrong during the boot process.

Now to launch my installer instance:

euca-run-instances -t m1.xlarge \
      -d "bootparams=ksdevice=link ip=dhcp vnc keymap=us lang=en_US console=ttyS0" \
      emi-BA8F405E

This boots into an interactive install, which listens for vnc connections. Note that due to the size of the initrd, this instance needs a significant amount of RAM; I used 2GB, but 1GB would have worked. Before proceeding, I attach the volume (which I could have done via block device mapping):

euca-attach-volume -i i-447E3E89 -d sdd vol-14AE3F68


I check euca-describe-instances for the instance's IP address, connect to it with a vnc client, and proceed with the install. Once the install completes, I detach the volume and terminate the instance:

• euca-detach-volume vol-14AE3F68
• euca-terminate-instances i-447E3E89

Finally, I convert the volume to a snapshot and register it:

• euca-create-snapshot vol-14AE3F68
• euca-register -n f16-test -s snap-2CBB42D9

I boot an instance of my new EMI, and ... it fails to have a network. There were multiple problems with the networking configuration:

1. The MAC address is hard-coded.
2. The device name has changed from eth0 to eth1 (maybe related to #1)
3. The NIC is configured to be controlled by NetworkManager

This is when I'm happy to have a vnc connection provided at the libvirt layer to debug the instance.  A quick setup of ifcfg-eth1 and a restart of the network gives me connectivity, and I'm up and running with a Fedora 16 instance installed entirely in the cloud.

The whole process took me about an hour or so this morning (not counting writing the xsl and shell script yesterday), and I imagine that the process would be much faster for subsequent attempts, and even faster when a kickstart is used.  Still, I'm not convinced that an approach like this has significant value over something like BoxGrinder or ami-creator.  Let the debate begin!  :-)

So last weekend I thought of trying out the Eutester project which has been up on projects.eucalyptus.com for a while and now the code been moved to github with some serious development happening. Well for those who are new to Eutester, it is a framework written in python which helps you test your Eucalyptus private [...]

Loving the work that the RENCI folks at Duke are doing on top of Eucalyptus. They’ve got a set of patches that sit atop Eucalyptus proper, and they call their patches “Neuca”. I lol’d when I found that out.  It rolls swimmingly off the tongue.

We’ve seen quite a few of these kinds of projects.  It’s a key indicator of success that people are building this functionality on top of our base.

Our ability to incorporate these kinds of patches directly into mainline will be a key indicator of our maturity as an open source company going forward.

As I discussed at SCALE 10x this past weekend, I believe that our current contributor agreement needs an update.  That work will take some time, and I can’t really say much about it yet — but the prospect of working more closely with RENCI and others provides strong motivation to Get It Right.  It’s a key challenge, and I’m exciting about tackling it head-on in the coming weeks.

(p.s. RAGE HATE SPELLING FAIL.  The likelihood of my typing “eucalytpus” is pretty much an even money bet.)

I want to talk about something new we’re working on at Eucalyptus, but first let me start with a little background. Quite simply, it is a hassle to get an image installed. The current process for Eucalyptus (as we document it) is to download a tarball, untar it, bundle/upload/register the kernel/ramdisk and image itself. That’s about 11 steps. We thought there must be a simpler way to do this.

What we came up with is eustore. In the spirit of euca2ools (euca- and euare- commands), eustore commands give  you access to a Eucalyptus image store. That’s store, as in storehouse, not a shop. We have some updated “base” images available on our servers. We have a catalog file that contains metadata about those images. The eustore tools simply give you access to those, and let you issue a single command to download an install an image on your local cloud (or any Eucalyptus cloud you have access to).

The code has been checked in with the euca2ools. To install and use the commands, you’ll need to build from source and tweak the setup.py. Let’s go over that now.

If you don’t have bzr, you’ll need to download it and grab the code with

bzr branch lp:euca2ools

You’ll find the eustore commands in euca2ools/commands/eustore. The commands still need to be added to setup.py, as does the package to get it installed with the rest of euca2ools. Here’s s patch script you can apply with “patch -p0 <setup.patch” (assuming you copy this into a file named setup.patch);

--- setup.py 2012-01-20 17:17:48.000000000 -0800
+++ setup.py 2012-01-20 17:18:53.000000000 -0800
@@ -161,10 +161,13 @@ setup(name = "euca2ools",
 "bin/euca-unbundle",
 "bin/euca-unmonitor-instances",
 "bin/euca-upload-bundle",
- "bin/euca-version"],
+ "bin/euca-version",
+ "bin/eustore-describe-images",
+ "bin/eustore-install-image"],
 url = "http://open.eucalyptus.com",
 packages = ["euca2ools", "euca2ools.nc", "euca2ools.commands",
- "euca2ools.commands.euca", "euca2ools.commands.euare"],
+ "euca2ools.commands.euca", "euca2ools.commands.euare",
+ "euca2ools.commands.eustore"],
 license = 'BSD (Simplified)',
 platforms = 'Posix; MacOS X; Windows',
 classifiers = [ 'Development Status :: 3 - Alpha',

Once that file is patched, installing euca2ools (+eustore) is as simple as running (as root)

python setup.py install

Once you do this, you’ll have access to 2 new commands; eustore-describe-images and eustore-install-image. Here are the command summaries;

Usage: eustore-describe-images [options]

Options:
 -h, --help show this help message and exit
 -v, --verbose display more information about images

 

Usage: eustore-install-image [options]

Options:
 -h, --help show this help message and exit
 -i IMAGE_NAME, --image_name=IMAGE_NAME
 name of image to install
 -b BUCKET, --bucket=BUCKET
 specify the bucket to store the images in
 -k KERNEL_TYPE, --kernel_type=KERNEL_TYPE
 specify the type you're using [xen|kvm]
 -d DIR, --dir=DIR specify a temporary directory for large files
 --kernel=KERNEL Override bundled kernel with one already installed
 --ramdisk=RAMDISK Override bundled ramdisk with one already installed

eustore-describe-images list the images available at emis.eucalyptus.com. You have the ability to change the url (using the EUSTORE_URL environment variable which is helpful sometimes). The output looks like this;

centos-x86_64-20111228 centos x86_64 2011.12.28 CentOS 5 1.3GB root
centos-x86_64-20120114 centos x86_64 2012.1.14 CentOS 5 1.3GB root
centos-lg-x86_64-20111228centos x86_64 2011.12.28 CentOS 5 4.5GB root
centos-lg-x86_64-20120114centos x86_64 2012.1.14 CentOS 5 4.5GB root
debian-x86_64-20111228 debian x86_64 2011.12.28 Debian 6 1.3GB root
debian-x86_64-20120114 debian x86_64 2012.1.14 Debian 6 1.3GB root
debian-lg-x86_64-20111228debian x86_64 2011.12.28 Debian 6 4.5GB root
debian-lg-x86_64-20120114debian x86_64 2012.1.14 Debian 6 4.5GB root
ubuntu-x86_64-20120114 ubuntu x86_64 2012.1.14 Ubuntu 10.04 1.3GB root
ubuntu-lg-x86_64-20120114ubuntu x86_64 2012.1.14 Ubuntu 10.04 4.5GB root

To install one of these images on your local cloud, you’d use eustore-install-image like this;

eustore-install-image -i debian-x86_64-20120114 -b myimages

This command installs the image named into the myimages bucket on the cloud you are setup to talk to. As with all euca2ools, you’d first source the eucarc file that came with your cloud credentials. I should point out something about uploading kernel and ramdisk to your cloud. Only the admin can install these. If you have admin credentials, the above command will work fine. If you don’t and want to install an image anyway, you would use the –kernel and –ramdisk options to refer to a kernel id and ramdisk id already installed on the cloud. That way, this command will ignore the kernel and ramdisk bundled with the image and refer to the previously uploaded ones.

The project management is happening here: https://projects.eucalyptus.com/redmine/projects/eustore/

It is discussed during the images meetings on IRC  (calendar here)

In my last entry, I explained how to checkout eucalyptus 3-devel and build it from source on Fedora 16.  This entry will explain how to follow that process with configuration and initialization of a single node cloud.

1) Configure environment variables.

export EUCALYPTUS=/opt/eucalyptus
export PATH=$PATH:$EUCALYPTUS/usr/sbin

2) Configure eucalyptus.conf -- Since this is a single node install on a network with DHCP, I am using SYSTEM mode for networking, which is the default.

USE_VIRTIO_DISK="1"
USE_VIRTIO_NET="1"
INSTANCE_PATH="/opt/eucalyptus/instances"
VNET_BRIDGE="br0"


3)  Set up proper file and directory permissions in the installed tree:

su -c "euca_conf --setup"
 
4) Initialize the database:

euca_conf --initialize

5) Create a bridge device and associate your primary NIC (this is specific to SYSTEM mode):

/etc/sysconfig/network-scripts/ifcfg-br0:
DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no



/etc/sysconfig/network-scripts/ifcfg-em1:
DEVICE="em1"
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
then restart your network

6) Get credentials and source them:

euca_conf --get-credentials admin.zip
source eucarc

7) Start the cloud components and register services:

su -c "/opt/eucalyptus/etc/init.d/eucalyptus-cloud start"

At this point, you should have a running cloud.  To verify the components:

euca-describe-walruses ; euca-describe-storage-controllers ; euca-describe-clusters

You should see something like:

WALRUS    walrus    walrus             192.168.51.251     ENABLED    {}
STORAGECONTROLLER    PARTI00  SC_251   192.168.51.251     ENABLED    {}
CLUSTER    PARTI00            CC_251   192.168.51.251     ENABLED    {}

And to ensure that the node controller is advertising resources:

euca-describe-availability-zones verbose

which shows:

AVAILABILITYZONE    PARTI00    192.168.51.251 arn:euca:eucalyptus:PARTI00:cluster:CC_251/
AVAILABILITYZONE    |- vm types    free / max   cpu   ram  disk
AVAILABILITYZONE    |- m1.small    0004 / 0004   1    128     2
AVAILABILITYZONE    |- c1.medium    0002 / 0002   1    256     5
AVAILABILITYZONE    |- m1.large    0001 / 0001   2    512    10
AVAILABILITYZONE    |- m1.xlarge    0000 / 0000   2   1024    20
AVAILABILITYZONE    |- c1.xlarge    0000 / 0000   4   2048    20

That's all for my second post.   Comments and corrections welcome.  See you on #eucalyptus !

Last time, we talked about a way to save some IAM resources on a cloud to a text file as a way to backup this information. We captured accounts/users/groups and policies. This post will focus on using the backup we created to restore those same resources to a new cloud (or the same one in a recovery scenario).

Since I’ve become fond of Python for some types of scripting, I decided I’d use python here to parse the text file. Originally, I thought I’d have python execute the euare- commands itself. Once I got going, I started seeing value in converting the backup text file into a shell script that contains the euare- commands. Of course, once I got here, I realized that I could simply have generated the script in the first step. Ah, well.. water under the bridge. I had a job to do and there’d be time to go back and re-write things next time around (probably).

We’ll use the same example set of resources from the last post:

accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
endgroup
endaccounts

The script I wrote is pretty brute force. It even has the name of the input file hard-coded! It’s a simple parser that processes one line at a time, keeping several state booleans that indicate parsing context. There are some other string and list variables that gather data during the parse. When enough data is parsed, euare- commands are emitted to standard out.

#!/usr/bin/env python

def main():
 inAccounts = False
 inUser = False
 inGroup = False
 inPolicy = False

accountName = None
 userName = None
 groupName = None
 policyName = None
 userList = []
 policyLines = []

f = open('post.iam', 'r')
 line = f.readline()
 while (line != ""):
 line = line.strip()
 idx = line.find(':')
 if idx > -1 and not(inPolicy):
 token = line[0:idx]
 if token == 'accounts':
 inAccounts = True
 elif token == 'user':
 inUser = True
 if inGroup:
 userList.append(line[idx+1:].strip())
 else:
 userName = line[idx+1:].strip()
 elif token == 'group':
 inGroup = True
 groupName = line[idx+1:].strip()
 elif token == 'policy-name':
 policyName = line[idx+1:].strip()
 elif token == 'policy-val':
 policyLines.append(line[idx+1:].strip())
 inPolicy = True
 elif line == 'enduser':
 #print "create user: "+userName+" for account "+accountName
 if userName != 'admin':
 print "euare-usercreate -u "+userName+" --delegate="+accountName
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-useruploadpolicy --delegate="+accountName+" -u "+userName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inPolicy = False
 inUser = False
 userName = None
 elif line == 'endgroup':
 #print "create group: "+groupName+" for account "+accountName +" with users;"
 print "euare-groupcreate -g "+groupName+" --delegate="+accountName
 for u in userList:
 print "euare-groupadduser --delegate="+accountName+" -g "+groupName+" -u "+u
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-groupuploadpolicy --delegate="+accountName+" -g "+groupName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inGroup = False
 inPolicy = False
 groupName = None
 userList = []
 elif line == 'endaccounts':
 inAccounts = False
 accountName = None
 else:
 if inAccounts and not(inUser) and not(inGroup) and not(inPolicy):
 accountName = line.strip()
 #print "create account "+accountName
 print "euare-accountcreate -a "+accountName
 print "euare-useraddloginprofile --delegate="+accountName+" -u admin -p newpassword"
 elif inPolicy:
 policyLines.append(line.strip())

 line = f.readline()

if __name__ == "__main__":
 main();

When I run this on the input file, it produces;

euare-accountcreate -a my account
euare-useraddloginprofile --delegate=my account -u admin -p newpassword
euare-usercreate -u testuser --delegate=my account
euare-useruploadpolicy --delegate=my account -u testuser -p testpolicy -o "{\"Statement\":[{\"Effect\": \"Allow\",\"Action\": \"*\",\"Resource\": \"*\",}]}"
euare-usercreate -u dev1 --delegate=my account
euare-usercreate -u dev2 --delegate=my account
euare-groupcreate -g developers --delegate=my account
euare-groupadduser --delegate=my account -g developers -u dev1
euare-groupadduser --delegate=my account -g developers -u dev2
euare-groupuploadpolicy --delegate=my account -g developers -p allow_all -o ""

You’ll notice the euare-addloginprofile commands, which simply set a default password.

Well it’s been really really long since I updated this space but looks like the right time to change a few things around starting with a new post in a new year! This post basically summarizes 2011 for me and ends with the latest news which comes with the new year. For a better understanding [...]

It’s always nice to visit family for a while, and see how the kids have grown.  FUDCon Blacksburg felt an awful lot like a family reunion — except one with a lot more learnin’ going on.  Just a small part of what I learned:

* ARM is coming.  Raspberry Pi is cool, and there’s way cooler down the road.  We’re working on our first little event kit that Eucalyptians will be able to use for demos at some point.  Right now it’s three laptops.  In a couple of years, it’s likely to be a laptop and a half-dozen itty bitty ARM systems.

* Talked with Seth and Smooge about a use case we’d been considering in Fedora-land for a long time: the “community cloud”, in which community members can basically dedicate machines to the Fedora cause.  It’s been a dream scenario for a long time — maybe we’ve got a shot at making it reality with Euca’s help.  Each contributed machine could become the equivalent of an “availability zone”, essentially.  Could be an interesting way of provisioning a lot of build systems in a pinch, and so on.  We’ll see how things shake out.

* Silvereye, a project to simplify Euca installation, is coming along.  I installed my first ever Eucalytpus system this weekend using Silvereye 0.01 and Euca 2.0.3, and I’m basically the equivalent of a trained chimpanzee.  We’ve got some improvements to make, but I’m convinced that we’re on the right path.  (Silvereye == Silver I == AgI == Silver Iodide == Cloud Seeder.)

* I love the ideas behind Boxgrinder, and Marek’s presentation reinforced that lovin’ feeling.  I think we should absolutely be pimping Boxgrinder as one of the primary tools to build Euca images.  Poke to Marek to make slide deck public plz. 

* Thanks to John Mark’s excellent Gluster presentation, I learned why Eucalyptus calls its storage mechanism “Walrus”.  I am clearly late to this understanding.

Good times, good times.  Looking forward to the next one, whenever and wherever that may be.

Amazon has supported the IAM (Identity and Access Management) API for some time now. The new release of Eucalyptus adds IAM support and got me thinking of how somebody could backup IAM settings. What really got me going on an actual solution was the need to save a set of accounts/users/groups and policies in order to restore them to a new Eucalyptus cloud.
My plan was to generate a data file which contains all of the information and can be parsed fairly easily to restore the information to the destination cloud. I considered writing JSON, but I had some time constraints and didn't feel like fiddling around getting the formatting just so. I chose to output some tokens followed by a colon. It looks like this:
accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
endgroup
endaccounts


I decided to write a bash script to run the commands and parse their output to produce the above tagged format. What you see below is what I came up with. It assumes you have environment variables set up properly (source the eucarc file). It loops through all accounts, then within each account, all users and groups. For users, it also looks for policies. For groups, it lists the users in the group and looks for policies.
#!/bin/bash
echo "accounts:"
for i in `euare-accountlist |awk '{ print $1 }'`
do
 echo $i
 for u in `euare-userlistbypath --delegate=$i`
 do
 u2=`echo $u |cut -d/ -f2-`
 u3=`basename $u2`
 echo user: $u3
 if [ `euare-userlistpolicies -u $u3 --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-userlistpolicies -u $u3 --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-usergetpolicy -u $u3 -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo enduser
 done

for j in `euare-grouplistbypath --delegate=$i | tail -n +2`
 do
 k=`echo $j |cut -d/ -f2-`
 l=`basename $k`
 echo group: $l
 for gu in `euare-grouplistusers -g $l --delegate=$i | tail -n +3`
 do
 gu2=`echo $gu |cut -d/ -f2-`
 gu3=`basename $gu2`
 echo user: $gu3
 done
 if [ `euare-grouplistpolicies -g $l --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-grouplistpolicies -g $l --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-groupgetpolicy -g $l -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo endgroup
 done
done
echo endaccounts


In the next post, I'll talk about how I used this backup data to restore accounts to a new cloud.

Note: AWS has now started their own list of API endpoints here. You may want to begin using that list as the definitive reference.



Another Note:  I am now collecting and publishing this information as JSON data. I am generating the HTML below from this JSON data.


Guy Rosen (@guyro on Twitter) recently asked about a comprehensive list of AWS service endpoints.  This information is notoriously difficult to find and seems to be spread across many different documents, release notes, etc.  Fortunately, I had most of this information already gathered together in the boto source code so I pulled that together and hunted down the stragglers and put this list together.

If you have any more information to provide or have corrections, etc. please comment below.  I'll try to keep this up to date over time.

Auto Scaling

• us-east-1: autoscaling.us-east-1.amazonaws.com
• us-west-1: autoscaling.us-west-1.amazonaws.com
• us-west-2: autoscaling.us-west-2.amazonaws.com
• sa-east-1: autoscaling.sa-east-1.amazonaws.com
• eu-west-1: autoscaling.eu-west-1.amazonaws.com
• ap-southeast-1: autoscaling.ap-southeast-1.amazonaws.com
• ap-northeast-1: autoscaling.ap-northeast-1.amazonaws.com

CloudFormation

• us-east-1: cloudformation.us-east-1.amazonaws.com
• us-west-1: cloudformation.us-west-1.amazonaws.com
• us-west-2: cloudformation.us-west-2.amazonaws.com
• sa-east-1: cloudformation.sa-east-1.amazonaws.com
• eu-west-1: cloudformation.eu-west-1.amazonaws.com
• ap-southeast-1: cloudformation.ap-southeast-1.amazonaws.com
• ap-northeast-1: cloudformation.ap-northeast-1.amazonaws.com

CloudFront

• universal: cloudfront.amazonaws.com

CloudWatch

• us-east-1: monitoring.us-east-1.amazonaws.com
• us-west-1: monitoring.us-west-1.amazonaws.com
• us-west-2: monitoring.us-west-2.amazonaws.com
• sa-east-1: monitoring.sa-east-1.amazonaws.com
• eu-west-1: monitoring.eu-west-1.amazonaws.com
• ap-southeast-1: monitoring.ap-southeast-1.amazonaws.com
• ap-northeast-1: monitoring.ap-northeast-1.amazonaws.com

DevPay

• universal: ls.amazonaws.com

ElastiCache

• us-east-1: elasticache.us-east-1.amazonaws.com
• us-west-1: elasticache.us-west-1.amazonaws.com
• eu-west-1: elasticache.eu-west-1.amazonaws.com
• ap-southeast-1: elasticache.ap-southeast-1.amazonaws.com
• ap-northeast-1: elasticache.ap-northeast-1.amazonaws.com

Elastic Beanstalk

• us-east-1: elasticbeanstalk.us-east-1.amazonaws.com

Elastic Compute Cloud

• us-east-1: ec2.us-east-1.amazonaws.com
• us-west-1: ec2.us-west-1.amazonaws.com
• us-west-2: ec2.us-west-2.amazonaws.com
• sa-east-1: ec2.sa-east-1.amazonaws.com
• eu-west-1: ec2.eu-west-1.amazonaws.com
• ap-southeast-1: ec2.ap-southeast-1.amazonaws.com
• ap-northeast-1: ec2.ap-northeast-1.amazonaws.com

Elastic Load Balancing

• us-east-1: elasticloadbalancing.us-east-1.amazonaws.com
• us-west-1: elasticloadbalancing.us-west-1.amazonaws.com
• us-west-2: elasticloadbalancing.us-west-2.amazonaws.com
• sa-east-1: elasticloadbalancing.sa-east-1.amazonaws.com
• eu-west-1: elasticloadbalancing.eu-west-1.amazonaws.com
• ap-southeast-1: elasticloadbalancing.ap-southeast-1.amazonaws.com
• ap-northeast-1: elasticloadbalancing.ap-northeast-1.amazonaws.com

Elastic Map Reduce

• us-east-1: elasticmapreduce.us-east-1.amazonaws.com
• us-west-1: elasticmapreduce.us-west-1.amazonaws.com
• us-west-2: elasticmapreduce.us-west-2.amazonaws.com
• sa-east-1: elasticmapreduce.sa-east-1.amazonaws.com
• eu-west-1: elasticmapreduce.eu-west-1.amazonaws.com
• ap-southeast-1: elasticmapreduce.ap-southeast-1.amazonaws.com
• ap-northeast-1: elasticmapreduce.ap-northeast-1.amazonaws.com

Flexible Payment Service

• sandbox: authorize.payments-sandbox.amazon.com/cobranded-ui/actions/start
• production: authorize.payments.amazon.com/cobranded-ui/actions/start
• sandbox: fps.sandbox.amazonaws.com
• production: fps.amazonaws.com

Identity & Access Management

• universal: iam.amazonaws.com

Import/Export

• universal: importexport.amazonaws.com

Mechanical Turk

• universal: mechanicalturk.amazonaws.com

Relational Data Service

• us-east-1: rds.us-east-1.amazonaws.com
• us-west-1: rds.us-west-1.amazonaws.com
• us-west-2: rds.us-west-2.amazonaws.com
• sa-east-1: rds.sa-east-1.amazonaws.com
• eu-west-1: rds.eu-west-1.amazonaws.com
• ap-southeast-1: rds.ap-southeast-1.amazonaws.com
• ap-northeast-1: rds.ap-northeast-1.amazonaws.com

Route 53

• universal: route53.amazonaws.com

Security Token Service

• universal: sts.amazonaws.com

Simple Email Service

• us-east-1: email.us-east-1.amazonaws.com

Simple Notification Service

• us-east-1: sns.us-east-1.amazonaws.com
• us-west-1: sns.us-west-1.amazonaws.com
• us-west-2: sns.us-west-2.amazonaws.com
• sa-east-1: sns.sa-east-1.amazonaws.com
• eu-west-1: sns.eu-west-1.amazonaws.com
• ap-southeast-1: sns.ap-southeast-1.amazonaws.com
• ap-northeast-1: sns.ap-northeast-1.amazonaws.com

Simple Queue Service

• us-east-1: sqs.us-east-1.amazonaws.com
• us-west-1: sqs.us-west-1.amazonaws.com
• us-west-2: sqs.us-west-2.amazonaws.com
• sa-east-1: sqs.sa-east-1.amazonaws.com
• eu-west-1: sqs.eu-west-1.amazonaws.com
• ap-southeast-1: sqs.ap-southeast-1.amazonaws.com
• ap-northeast-1: sqs.ap-northeast-1.amazonaws.com

Simple Storage Service

• : s3.amazonaws.com
• us-west-1: s3-us-west-1.amazonaws.com
• us-west-2: s3-us-west-2.amazonaws.com
• sa-east-1: s3.sa-east-1.amazonaws.com
• eu-west-1: s3-eu-west-1.amazonaws.com
• ap-southeast-1: s3-ap-southeast-1.amazonaws.com
• ap-northeast-1: s3-ap-northeast-1.amazonaws.com

SimpleDB

• us-east-1: sdb.amazonaws.com
• us-west-1: sdb.us-west-1.amazonaws.com
• us-west-2: sdb.us-west-2.amazonaws.com
• sa-east-1: sdb.sa-east-1.amazonaws.com
• eu-west-1: sdb.eu-west-1.amazonaws.com
• ap-southeast-1: sdb.ap-southeast-1.amazonaws.com
• ap-northeast-1: sdb.ap-northeast-1.amazonaws.com

Virtual Private Cloud

• us-east-1: ec2.us-east-1.amazonaws.com
• us-west-1: ec2.us-west-1.amazonaws.com
• us-west-2: ec2.us-west-2.amazonaws.com
• sa-east-1: vpc.sa-east-1.amazonaws.com
• eu-west-1: ec2.eu-west-1.amazonaws.com
• ap-southeast-1: ec2.ap-southeast-1.amazonaws.com
• ap-northeast-1: ec2.ap-northeast-1.amazonaws.com

Here’s a list of popular open source products that cannot currently be found in Fedora repos:

• Zimbra
• JasperSoft
• SugarCRM
• Alfresco
• Magento
• Eucalyptus
• JBoss

Once upon a time, it was part of my job to help these kinds of companies to work more closely with Fedora. We created the ISV SIG for this purpose. Karsten and I would go to trade shows and meet with various open source vendors, and we’d talk with them at length about the great benefit of leveraging the Fedora install base, and the power of “yum install YourCoolProduct”, and the general usefulness of building an ISV packaging community, and they’d nod and smile, and then we’d have a follow-up meeting or two to discuss the ins and outs of being in a distro. And then… well, nothing much would happen.

Now, as it turns out, I’m in a position to appreciate, and articulate, these issues from the ISV’s perspective.

What do the applications listed above have in common? A couple of key points.

Point One: they are all sponsored by companies, who use the open source projects as a base from which to build proprietary products.

Point Two: they all tend to be the primary application running on their machine — in other words, they are appliance stacks — and they need to limit variance in those stacks to help guarantee a good experience for their users.

It’s easy to claim, and many do, that these projects aren’t in Fedora (or Ubuntu, for that matter) because of Point One. In truth, Point Two is *way* more important.

There’s a great page on the Fedora Wiki that does a good job of discussing the potential gains and losses of putting your ISV application into Fedora. I’m going to go through those gains and losses, and share my opinions of them, now that I’m on the other side of the fenceline.

[GAIN] Reduced maintenance burden for all dependencies that are already packaged in Fedora: no need to ship security updates for those components.

This is a good potential gain, but note that it does not require the ISV to be *in* the distro to get this gain. It’s entirely possible to package *on top of* the distro, track the distro closely, and get all of these maintenance gains, without incurring the high cost of pushing packages into the distro and maintaining them. I suspect that this is precisely what many companies choose to do.

[GAIN] Code auditability: the Fedora packaging processes ensure that all code is described by metadata (i.e., spec files). The packaging tools allow this data to be queried in informative ways. ISVs don’t necessarily track this data otherwise.

Also true, but again, note that it’s possible to build RPMs and get the same advantages without putting those RPMs into the distro. There are two separate costs here: there’s the cost of building an RPM, which is comparatively low if you’ve got the source and an experienced packager at your disposal — but then there’s the cost of pushing the RPM into the distro and following the distro’s rigorous rules around versioning and namespacing and supportability, which is a *much* higher cost for the ISV. The gain from that additional cost must therefore be demonstrably compelling.

[GAIN] Availability of package-specific expertise: ISVs can consult other packagers about the upsteams of their dependencies. Each Fedora package maintainer acts as a known point of contact for their package’s upstream project.

This is very much a potential gain, if it’s true. But what happens when most of the packages aren’t yet in Fedora? This is especially problematic in the Java world, where there are tons and tons and tons of jar files that are not “packaged” as such in Fedora, but are still perfectly useful to the Java community in jar form. If the distro packaging expertise for a particular jar doesn’t yet exist yet, then the company who pushes the packages into the distro must take on the initial cost of becoming that expert. It’s definitely true that this expertise can be shared over time, and also true that such shared expertise is a long-term win — but the upfront cost is high, especially for a small company that has lots of competing priorities.

[GAIN] The trust of Fedora users: ISV products packaged in the Fedora way will be more warmly-received by Fedora users than standalone GNU/Linux binaries.

Citation needed. I mean, yes, I believe this too, but it’s a gain that’s difficult to quantify. The real benefit we’re trying to claim here is that “yum install foo” is a simpler and awesomer experience — and it is. But the difference between “yum install foo” and “wget foo-installer | sh”, which adds the ISV’s yum repo and gpg key and then kicks off “yum install foo”, is not really that great.

[GAIN] Stability on Fedora: standalone binaries break frequently because Fedora is such a fast-moving target. Built-from-source packages have proven much more stable, since incompatilities are caught during mass rebuilds.

This is a bit of a tautology. It’s essentially arguing that your ISV packages will build better with Fedora because you’re working to make them build better with Fedora. Which is true, but again, can be true by building *on top of* Fedora and not *in* Fedora.  And it also only addresses build time failures, which, for an application, are failures that you’re likely to find immediately anyway if you’re doing proper build/test integration internally.

[GAIN] Bug triaging: Fedora users report bugs to Red Hat Bugzilla first; the package maintainer decides if it’s a packaging bug or an upstream bug. If it’s an upstream bug the packager will ideally create a minimal test case and send it to the upstream maintainers.

This is a strong *potential* gain, if the package maintainer is a trusted and responsible member of the community. But what if the package maintainer is an employee of the company, as is usually the case? It’s not a gain at all.  And what if the package maintainer also maintains 20 other packages, and isn’t particularly responsive?  Then it’s a net loss.

[LOSS] Binary dependency predictability: dependency updates may mean that the deployed set of components is not the same set of binaries the ISV tested during their release process.

Bingo!  No more calls, please — we have a winner.

Here’s the thing: an ISV does not have the luxury of dealing with variance. We’re dealing with tons of bugs, every day, because we’re young companies, pushing as hard and as fast as we can to make our software experience better. When we’re trying to kill a crazy bug for users/customers, the first order of business is to reduce the uncertainties, and the easiest way to do that is to be *very* specific about configurations. This is especially true as the software increases in complexity.

We can assume high competence and good faith on the part of community maintainers, and still be relatively certain that those good actors will make changes, for good reasons, that will damage the ISV’s application stack in unpredictable and important ways. Software is mean-spirited like that.

This could, in theory, be mitigated by keeping multiple versions of things, and having better mechanisms for tracking those versions. This is something that Red Hat Network customers wanted for years, and finally got — the ability to install a very specific package manifest that is not “all latest packages”, but “these specific package versions”.  But Linux distros don’t work that way, for good or ill.

In theory, everyone should always be running the latest version of things. In practice, that can be very difficult — and it can be *especially* difficult for the ISV when multiple distros have different notions of what the latest version is, and *exceptionally* difficult when those package manifests can change without warning, and outside of your control.

Maintaining a functioning product in multiple cutting-edge distros, with different release cycles and different dependencies, requires a serious, serious commitment to continuous integration and testing. I believe that Eucalyptus has a better process for this than most — and still it will be a tremendous challenge for us to keep up with two different fast-moving distros in Fedora and Ubuntu.

[LOSS] Unity with Windows release process: someone on the ISV’s team will need to be a Fedora contributor or they will need to recruit an external packager.

You can replace “Unity with Windows release process” with “Unity with Ubuntu release process” and the problem is the same. There are huge differences, of course, between a Windows release process and a Linux release process — but even staying in the Linux world, there’s a considerable difference between the Ubuntu release process and the Fedora release process, and expertise in the one in no way guarantees success in the other.

[LOSS] Ability to customize dependencies arbitrarily: there are rare cases where Fedora ships different versions of the same component for compatibility but in general this is strongly discouraged; custom patches should be sent upstream or eliminated by patching the product’s code to not require them.

Absolutely.

[LOSS] Download counting/tracking: if an ISV provides a tar-based distribution from their website, they can track counts and/or emails. This may be important for their marketing department.

Ayup.

* * * * *

It looks pretty grim in the end, doesn’t it? Well, it’s not as dark as all that. There are legitimate ways for the committed ISV to bridge the gaps over time:

1. Commit to building RPMs (and dpkgs), from source, the right way, for the ISV product, and making those source packages available to whomever wants them. There are legitimate reasons for an open source company to do this, and it’s a necessary precondition to being in the distros anyway.

2. Release their Linux versions as add-on yum/dpkg repos.  Of course, this also means being able to supersede/obsolete distro packages with foo packages, but this is easily done by maintaining separate namespaces.

3. Continue to work with other ISV vendors on packaging best practices at every opportunity, even if those packages don’t immediately end up in the distro.

4. Explore development builds that depend on the latest packages, available from wherever. One of the great advantages of Fedora, and other fast-moving distros, is that they do a great job of managing the future. We don’t want to live in the future, but we certainly want to have our eye on it, and that’s a great reason to continue to *try* be in Fedora — but we also need to make it clear to potential users that the future and the present don’t always see eye-to-eye, and that can be difficult messaging to convey.

The truth of the matter is that not every user understands the intricacies of the open source development model, and most ISVs in a competitive market get one shot to connect with their potential customers. One. Which means that the ISVs are going to do everything they possibly can to make sure that they’ve got control over how that experience goes, at the lowest possible development cost.

Fedora can afford to live right on the bleeding edge because they’ve got CentOS/RHEL to fall back on. Not everyone has that luxury.

(p.s. looking forward to talking more about this at FUDCon.  Also: the drinking.)

Eucalyptus is now a proud sponsor of Bull City Coworking. Various folks have tried to get a coworking operation up and running in Durham in the last little while; props to Robert Petrusz and the gang for actually getting it done.

Both Andy Grimm and I are Eucalyptus employees who live and work in Durham.  Now we have a space to hang out in, which is handy, because while working from home has its advantages, if you do it *every single day of your life* it can get old in a hurry.  So here were are, in East Downtown Durham, a stone’s throw from the best Cuban food in the Triangle.  The space is spare, but growing.  I CAN HAZ WHITEBOARDS, which is awesome.

So, Durhamites.  If you want to get out of your stuffy home office and come hang out for the day sometime, and especially if you want to talk about open source cloud awesomeness, ping me and I’ll set you up with a day pass.  Y’all come, hear?

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 32,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 12 sold-out performances for that many people to see it.

Click here to see the complete report.

After getting some free time to put together a high-level diagram of the Varish/Walrus setup we are using at Eucalyptus Systems, I decided to use it as an opportunity to make it my first technical blog. The Inspiration Here at Eucalyptus Systems, we are really big on “drinking our own champagne“.  We are in the process [...]

I'll apologize up front for that horrible pun in the title.  No excuse, really.

After 18 months at Eucalyptus, the best private cloud vendor out there, I have decided to see what things are like on the public cloud side.  As of Monday, December 19, I will be a senior engineer at Amazon Web Services.

I was very reluctant to leave Eucalyptus.  It is a great company, full of great people and with a corporate culture that absolutely cannot be beat.  And, while a lot of people's attention has been focused on shiny new things over the past year, Eucalyptus has quietly and steadily built amazing sales, support, marketing and professional services teams to match their already awesome engineering team.  2012 is going to be another kick-ass year for Eucalyptus and I really hate to miss that.

But the idea of seeing how the sausage is made at the biggest public cloud is an opportunity I couldn't pass up.  In my new job, I will still be focusing on software tools and how to make it easier for developer's to use cloud infrastructures, both public and private.  I will still be doing a lot of Python stuff and definitely still making sure that boto stays a popular, useful and independent open source project just as it did while I was at Eucalyptus.

It should be fun!

12.13.11 | CloudTweaks | Written by: Muzaffar Ismail

As the end of the year for 2011 makes its inexorable approach, the forecasts for 2012, in particular for cloud computing are flying helter-skelter. Both IDC and Gartner have weighed in with their forecasts and many more continue to release their forecasts regarding cloud computing for next year. Key amongst these predictions is the fact that this time around many of them are saying that 2012 will mark the beginning of the cloud computing revolution. While some newcomers to the cloud bandwagon will gush with great gusto and enthusiasm about the upcoming landslide of cloud converts for next year, many have forgotten that nearly the same statement was given for 2011.

More seasoned Cloud watchers are as usual watching carefully. While consumerization and multiple device access may well drive more conversions to the Cloud, it is wise to note that most studies or surveys have revealed that 2011 conversions based on these reasons have never been overwhelmingly so.
<script type="text/javascript"></script>
It isn’t a bad thing to take all of these forecasts with a pinch of salt, not only so that you don’t end up over-extending yourself by end of 2012, but also because it may turn you and yours into a pioneer with no support and your ass very much uncovered. Covering your own ass is still the number 1 priority for most employees, even Cloud pioneers so swallow down that gusto and keep a watch for the Cloud pitfalls for 2012. Thankfully, some 2012 forecasts have revealed such pitfalls. Some of these forecast Cloud pitfalls include revelations such as:

• The public Cloud isn’t really green – While many proponents of Cloud Computing have mentioned that the Cloud offers great ‘green’ environment saving potential it is in actuality just another method of passing the buck. This is because all the public Cloud service operators must have some form of server or data storage themselves to create their Cloud solutions. As more people move to the Cloud in 2012 these public Cloud service providers may also have to increase the size of their data centres to accommodate these new Cloud adopters. More energy will be required to power these data centres which will attract attention one way or another. Eventually (very soon more like) everyone is going to catch on, especially as public Cloud service provider data centres continue to grow exponentially. Playing the ‘green’ card in your deck at this time may lead you to a pitfall you may not be able to climb out of, especially if your only excuse is feigning ignorance.

• A major Cloud security breach is expected and no one will know who to blame – Despite all the security trials and tribulations of 2011, an even larger security breach is expected in 2012, with particular focus on Cloud solutions and storage. While this is still pure speculation, the fact that many users of the Cloud as well as Cloud providers themselves are rather unclear about who should be watching the security for Cloud solutions and storage is even more troubling (based on the Security of Cloud Computing Providers Study). In the likely event that such a large security breach were to happen, a company and its IT administrator could be literally caught with their pants down. This could be even more devastating if there is insufficient backup procedures in place, so do make sure about this beforehand regardless of your position in the Cloud.

Click here to continue reading or visit www.cloudtweaks.com.

If you are an AWS user with EC2 instances running, you may have already gotten an email from AWS informing you that your instance(s) will be rebooted in the near future.  I'm not exactly sure what is prompting this massive rebooting binge but the good folks at AWS have actually provided a new EC2 API request just so you can find out about upcoming maintenance events planned for your instances.

We just committed code to boto that adds support for the new DescribeInstanceStatus request.  Using this, you can programmatically query the status of any or all of your EC2 instances and find out if there is a reboot in their future and, if so, when to expect it.

Here's an example of using the new method and accessing the data returned by it.


<script src="https://gist.github.com/1443559.js?file=gistfile1.py"></script>

Eucalyptus has a long history of listening to our community. You asked for Eucalyptus to have a life of its own, and Eucalyptus Systems was born. You asked for a different license (GPL) and you got it. You asked for easier ways to install Eucalyptus and so we packaged Eucalyptus for the main linux distributions, we delivered UEC and we created FastStart.

We are still listening to our community. Currently we are working on the Eucalyptus 3 feature set and development process. Check our roadmap for more information
on the feature set, and stay tuned for more information about this
important release.

Our mantra is and always will be:  "Listen to our community and deliver quality software".

We always strive to get better at it. We are now in the process of redesigning the way we communicate with you. Currently we differentiate our users between customers (paying users) and community (intended here as the non-paying users). We have two web sites (www and open), two different sets of communication -- a support team,  an issue tracker, phone line 24/7 and direct emails for customers and a forum, IRC, launchpad and mailing list for community.

This setup is common enough, yet in our case it seems a bit arbitrary. Our engineers, our sales people, and our support team help community members on IRC and in the forum. Conversely we are all dedicated to help our customers. So we are in the process of removing this artificial division. Our support team is working to integrate the forum, mailing lists, and issue tracker. Our sales and marketing teams are working closely to understand the needs of our users and to provide them the information they want. Our professional services team is producing online training material. Our web team is working to tie all of the above together in a seamless and user-friendly way. And our engineering team is readying the best release of Eucalyptus yet.

Eucalyptus 3 will bring a lot of changes for our community: reconciling the various source branches into one, bringing a new plug-in architecture, and opening up the development process. Before Eucalyptus 3 is released, our two web sites will merge. We'll group information together following the cloud IT roles. Our webinars, forum posts, technical information, and documentation will be focused on delivering the best possible service to the specific IT role.

The new web site redesign, will put our community in the driver's seat. We will present updates and new design soon, and your feedback is necessary to carry on this ambitious plan.

One Eucalyptus Community.

Skip Directly to [Instructions on How to Run the Video-Processing Prototype]

A Developer Walks through Cloud

1. Little Phone, Big Cloud

A few months ago, a phrase caught my attention: “Instagram for Video”. It was an interesting idea for a mobile application. As a software designer, I dug into the idea, soon to realize one major implementation challenge.

It turns out that video is a collection of pictures–many, many pictures. Given the standard 24 frames-per-second rate, even an one-minute-long video would be comprised of 1440 pictures, which meant image-processing of 1440 pictures on a mobile phone. That is a lot of pictures for a small battery in your mobile phone to handle.

There is an alternative way to the scenario; let’s consider moving the image-processing task over to a remote machine that is bigger, stronger, and meaner. In this scenario, the mobile phone could upload the video to a server via the internet, process it remotely, and retrieve the processed video back in a seamless fashion.

However, there is one absolutely-crucial requirement in this scenario; we are going to need a big, big, big machine–big enough to handle millions of requests once this killer application goes viral (go big or go home). There is only one answer to this type of demand: “the Cloud.”

Luckily, there is an open-source cloud available; Eucalyptus is an open-source Infrastructure-as-a-Service cloud platform whose APIs are compatible with the ones with Amazon’s EC2. This makes Eucalyptus an ideal in-house cloud application development platform. It guarantees that once my killer application runs on Eucalyptus, it will also run on EC2 with no modifications required, thus creating a truly portable cloud application with the world-wide deployability.

2. IaaS Cloud

For those who are not familiar with the IaaS clouds, let me to take you to a quick walkthrough to the cloud.

Eucalyptus and Amazon’s EC2 offer “Infrastructure-as-a-Service” cloud platforms. It means that a cloud-user can request, “Hey cloud, I need 5 machines with full network connectivity and access to the storage,” then within minutes, the user will have the complete system ready for use.

Take this concept little further; instead of requesting machines for generic purposes, the cloud-user could have specified which machines to serve as what purposes at the creation. For instance, using the example in this article, the cloud-user could have asked, “Hey cloud, I want one machine to work as a collector while the rest as image-processors, and have them process my cat video immediately!” Then, the cloud would have brought up a network of machines with the specific tasks assigned to each machine, and they would have worked on processing of the cat video right away. Once the processing was complete, the machines would have been self-terminated, leaving only the processed cat video behind.

3. App on the Cloud

Let’s go back to the video-processing application on the cloud. Here I will cover some major design considerations when developing applications on the cloud.

3.1. Parallelism and Elasticity

Designing an application on a distributed system requires a process to be broken down into small tasks. Then, one must identify the tasks that can bring parallelism into the process. In this video-processing application, the process can be broken down into 3 major steps: decoding the video into images, processing the images, and encoding the processed images back to a video. Given these breakdowns, the natural approach is to distribute the image-processing task over multiple machines and assign a single machine to perform the encoding and decoding.

One important characteristic of the cloud that you must realize at the core of the design is the elasticity of the cloud. The elasticity is what differentiates the cloud applications from the traditional distributed applications. Traditionally, in a distributed computing environment, the number of nodes N in the system is a static value that is unchangeable during a job. However, in the cloud environment, there is no bound in the number N, theoretically the number N is limitless. This means that at any given point during the job, the system should expect the number N to grow, or even shrink in some cases. For instance, in our video-processing application, we could initially start with 5 machines assigned to be image-processing nodes, however in the middle of the processing, we should be able to add 5 more nodes to boost the productivity. Taking such advantage of the elasticity must be considered at the design level of the application.

3.2. Prototype

Following is the overview of the prototype of the video-processing application in the cloud.

For more detailed instructions, please go to the page [Instructions on How to Run the Video-Processing Prototype]

The goal of the prototype is to demonstrate a cloud application that performs image-processing tasks in a distributed fashion. The application takes input of a video file, performs image-processing in parallel, and when it terminates, the processed video file is stored in a known storage location provided.

For the simplicity of the prototype, let’s assume that there is a machine that works as a file server that has an apache web-service running in the open, which is accessible from the cloud. In other words, any virtual instances(nodes) spawned on the cloud will have access to the files on the file server via download(wget). Given this setup, for instance, when we trigger the collector node, it can download the input video file from the file server to start the process.

For the prototype, we need to construct two types of nodes: collector node and image-process node. However, before I go into further details, I must explain what takes places when the cloud-user requests an instance from the IasS cloud.

When the cloud-user asks the cloud, “Hey cloud, I need one machine,” the user is required to specify the image of the machine. In other words, the cloud-user must request, “Hey cloud, I need one machine with the RHEL 6.1 image that I have prepared for this video-processing prototype.” Then, the cloud will bring up a virtual instance that is flashed with the specified RHEL 6.1 image. Since users can prepare and upload images of their choices to the cloud, the possibilities are limitless on what you want the instances to do or to become.

For this particular prototype, I prepared a single image that would be used by all nodes. I took a generic Ubuntu Karmic image as the base image and modified its ‘rc.local’ script, which is the default script that gets executed automatically when the image boots up. The modified ‘rc.local’ script is set to read a line from the ‘user-data’ field, which get passed to the instance from the cloud-user at the creation. This small modification allows me to control the rolls of the instances with having only one image. For example, I can request, “Hey cloud, I want one instance with my special Ubuntu image and have it run the script ‘collector.pl’”, then later, I can ask, “Hey cloud, I want another instance with the same image, but this one will run the script ‘processor.pl’.”

The requests in the example above would look like the below. Notice using the same image ID ‘emi-9BD01749′, but different ‘user-data’ values (-d).

First request to bring up a collector:

euca-run-instances emi-9BD01749 -k mykey0 -n 1 -g group0 -t c1.medium -d “collector.pl”

Second request to bring up a processor:

euca-run-instances emi-9BD01749 -k mykey0 -n 1 -g group0 -t c1.medium -d “processor.pl”

In the prototype, the actual requests contain more information than just a script name. The first request looks like,

euca-run-instances emi-9BD01749 -k mykey0 -n 1 -g group0 -t c1.medium -d “collector.pl 192.168.7.77 [lovemycat.avi]“

This command translates to, after the instance boots up, it downloads the specified script ‘collector.pl’ from the file server at ’192.168.7.77′ via wget and execute the script. The purpose of the script ‘collector.pl’ is to turn the instance into the collector node for the video-processing application. First, the script installs all the necessary softwares via apt-get commands in Ubuntu; it uses various open-source softwares for the encoding and decoding tasks. It also installs the NFS server to create a shared directory where the processing nodes can access. Second, it downloads the target video file ‘lovemycat.avi’ from the file server at ’192.168.7.77′ (for the convenience of the prototype, the file server is designed to provide all the external file resources to the instances). Then, the collector node decodes the avi file into a collection of JPEG images. These image files are stored in the shared directory opened up by the NFS server. Now, the collector node waits for the image files to be processed by the processing nodes. The collector node’s job is to periodically scan the shared directory for the progress.

After the collector node enters the stage where it idles and scans, the next step is to start a group of the processing nodes by requesting,

euca-run-instances emi-9BD01749 -k mykey0 -n 3 -g group0 -t c1.medium -d “processor.pl 192.168.7.77 [10.219.1.2 neon.scm]“

As result, 3 instances will boot up, download the specified script ‘processor.pl’ from the file server at ’192.168.7.77′, and convert themselves into the image-processing nodes. It installs the opens-source image-processing software GIMP and the NFS client. It performs NFS-mount to the shared directory of the collector node, whose IP is at ’10.219.1.2′. Then, these processing nodes will start picking up image files from the shared directory and perform image-processing using GIMP according to the script ‘neon.scm’.

The syntax of the user-data for this image is:

-d “<script> <file_server_IP> [ <arguments_for_script> ]“.

Now, here is one crucial design decision that compliments the elasticity of the cloud. The work-unit for the image-processing is set to be 20 images at a time. This means that each node is only allowed to grab a chunk of 20 images at a time to perform image-processing. Under this policy, the processing nodes must frequently inquire the collector node for a small amount of work, instead of pre-determining the complete workload for each processing node prior to the beginning of the processing. This approach allows more processing nodes to be added to the system at any moment, thus taking full advantage of the elasticity.

When the processing nodes discover that there are no more images to be processed, they will be self-terminated, freeing up the computing resources for the cloud. When the collector node learns that all the images have been processed, it wakes up and encodes the images to a new video file. The final AVI file will be uploaded to the storage location belongs to the cloud-user. Eucalyptus and EC2 offer S3 storage units that allow such operation, however I will skip the details for later.

This prototype demonstrates how a complex operation, such as distributed video-processing, can be automated using the cloud. However, the automation is just a tip of the iceberg for the cloud. The raw power of the cloud comes from the ability to instantly replicate the application in a massive scale across the world. Such capability of the cloud contributes to the recent booming development in Software-as-a-Service (SaaS) solutions.

Extra. Links to Processed Videos

Using Invert Filter -

• Drive: http://youtu.be/pLkzIWy6_9Q
• Teapot: http://youtu.be/XujIfTtSGxE
• Beach: http://youtu.be/fIm-tlnf_D4

Using Edge Filter -

• Drive: http://youtu.be/tG92j1edFR8
• Teapot: http://youtu.be/LZhdbavlySc

Using Motion Blur Filter -

• Beach: http://youtu.be/FD4bDgoBCMg

Related. Links to Project Home Page -

• Video-Processing Cloud Application Project
• Instructions on How to Run the Video-Processing Prototype

One thing I can say for sure: there is no time to get bored here at Eucalyptus. If you like big challenges and you have it to take them on, consider joining our team: I'm extremely pleased to say that Greg just did so.

If you are at Cloud Expo in Santa Clara or in the Bay area, come and meet Greg (or Rich or Marten or me for that matter :) at EucaSocial, or visit us at our booth (#615).

I have been knowing Greg by fame for a while, but in person for only few months, and I can say that the energy, and passion he brings to Eucalyptus are intoxicating. You may have already noticed the impulse on our community activities (from our #eucalyptus IRC channel, our mailing lists, our projects and our planet) and this is just the start!

The only downside is that I don't have anymore the exclusive '-g' at the end of emails. Welcome Greg:   I'm  thrilled to be able to work with and learn from you.

Don’t miss out on the only enterprise IT event in 2011 covering the entire scope of cloud computing. While you are there stop by booth 868 and chat with CoreSite and Interxion!

When: November 7-10
Where: Santa Clara Convention Center, 5001 Great America Pkwy, Santa Clara, CA 95054

Cloud Expo Guest of Interxion & CoreSite Exhibitor (Booth 868) “VIP Gold” Pass Registration 3 Easy Steps:

1. Click here for our special registration page
2. Enter the Coupon Code: interxionVIPgold [case sensitive]
3. The price will re-set from $2,000 to $0, and you can then complete the brief registration process for full access to all sessions, all days and all tracks

(For these complimentary VIP Gold passes, Luncheon, access to the Cloud Computing Bootcamp and the Collectible Conference Bag are NOT included.)

You already know our <href>planet, since quite a few visit it regularly and requested to add feeds (let us know if you want your feed on our planet). When you visit it, you will find a somewhat ordinary planet site, perhaps too plain if you will, but what I want to mention in this blog, is how we are handling it and how it is running 'in the cloud'.

In the previous drinking champagne blog, I mentioned we have quite a few application and services in our internal production cloud (of course powered by Eucalyptus) and planet is one of them. What we did with planet, is to make it very simple to deploy and customize it: the original work was done by Mark Atwood. In order to take advantage of the cloud we heavily relied on the meta-data service.

The meta-data service allows for instances started in the cloud (private and public clouds which follows the AWS API) to learn data pertinent to the instance itself (hence it is called meta-data). Public IP, ssh keys, storage information, instance IDs (EMIs, kernel and ramdisk) are examples of what can be retrieved. All these data are accessible at http://169.254.169.254/latest/meta-data, and are easily accessible from within the instance with a browser or most likely with wget or curl.

Amongst these data, the user is allowed to pass few kB of data to the instance. To do so one could use the euca2ools and in particular the euca-run-instances with the -f or -d option. The instance can then access this data at http://169.254.169.254/latest/user-data: cloud-init uses it to run scripts at boot time or otherwise customize the instance and the official Amazon Linux AMI uses a port of cloud-init. Cloud-init is not yet available to all distros and earlier version of Eucalyptus suffered a bug which prevented cloud-init to work properly (instance would get delayed at boot time), so we decided to use a much simpler rc.local script to allow for a subset of its functionality. You can find information on the rc.local script we use and of other information about images on projects.eucalyptus.com. With these starter images in our production cloud, we set to host planet in an instance.

Our first attempt to run a service in the cloud, emulated the boot from EBS capability (see our issue tracker in the cloud), but this time we changed completely tack. We pushed all the planet's configuration into a Walrus bucket, then we created a script to be used when starting an instance.

You can inspect the script on projects.eucalyptus.com under the Cloud Application Architect area. The script gets all the css, ini and others needed file from a walrus bucket, installs nginx and other needed packages, and set up a cron-job to re-read the configuration files at set intervals, thus allowing for the dynamic configuration of planet. When we changed the logo, it was a matter of uploading the new css and png files, and presto! Planet got a new skin.

While this setup seems complicated at first glance, it is fairly easy: consider that it's all done with a few line of a shell script. It also allows for a easy failure recovery since restarting planet is a matter of 2 euca2ools command. If the instance were to fail we can issue:
</href="http:>

<href> euca-run-instances -f planet.sh -k my-ssh-key emi-F3DF1488</href="http:>
<href> euca-associate-address -i i-xxxxx 173.205.188.124</href="http:>
<href></href="http:>

<href>
and since there is no persistent data on the instance, that's all we need to do. And to apply a security update, we simply spin up a new instance with the same script (the script we use upgrades to all latest packages at start up), disassociate the public IP from the old instance and associate it to the new instance. And of course we can terminate the old planet once the elastic IP has been moved.

To customize the planet we upload the new version of the specific file to the Walrus bucket, and to do so we use a version of s3curl modified to allow for different endpoint then S3. For example to add a new feed we first get the current planet.ini:
</href="http:>

<href> wget http://173.205.188.8:8773/services/Walrus/planet/planet.ini</href="http:>
<href></href="http:>

<href>
modify it, to add the new feed, and we upload it back into the bucket</href="http:>

s3curl --id graziano --acl public-read --put planet.ini -- http://173.205.188.8:8773/services/Walrus/planet/planet.ini

<href></href="http:>

<href>
and wait for the cron-job to execute. Easy, isn't it?

We have few more scripts which we use for our production services on projects.eucalyptus.com and if you have a similar script you want to share, let us know and we'll add it, or do a git-hub merge request. The recipes are ready to go, just add Eucalyptus to it.</href="http:>

Last week I wrote a bit about the extent to which Open Source software has become accepted - and in fact an expectation - in many Enterprise Infrastructure stacks, and particularly so when it comes to IaaS Cloud computing:

http://open.eucalyptus.com/blog/2011/10/17/world-flat

This evening while I was hopping from link to link reading about KVM's slow but steady inroads into the hypervisor market, and the oVirt project I came across an interesting article that shared a lot of my points, but was slightly broader:

http://www.linuxinsider.com/story/Opening-the-Door-to-Innovation-73448.html

While I think the whole piece is well thought out and worth a read, I felt the author really hit the nail on the head, at gets at the heart of the point I was making last week:

"In today's modern computing environment, there's even more incentive to go the open innovation route: integration with and inclusion in other projects. Considering how many software components form the basis for data center automation, cloud computing and all virtualization, it's rare that a new technology could infiltrate data center infrastructure without being open source.

read more

The best part about living in Seattle is having access to Amazon Fresh for grocery shopping, with delivery right to my apartment. The level of convenience and quality has been great, especially for someone like me who is absolutely terrible at making time for Normal Life Activities, viz. shopping, having a driver’s license that matches the state in which I live, taking clothes to the dry cleaners, and diverse other errands and chores that aren’t related to working.

As of October 1, I am pleased to report that I have the extremely coveted Big Radish status, which is currently my favorite frequent-buyer-program name.

Big Radish status: giving renewed (and voluntary!) meaning to the old phrase, “I owe my soul to the company store”.

Filed under: Amazon

For those of you versed in Socioeconomic analysis, yes - I am borrowing the title of my blog post from Thomas Friedman's book of the same title. Personally, I preferred 'The Lexus and the Olive Tree' by Friedman, but then again this might be because I grew up in the Middle East and saw the subject matter unfolding first hand.

Anyway, enough about politics. But, it has been very interesting to see how Open Source software has done its part to 'flatten' the spaces between individuals, software engineers, infrastructure engineers, architects, and just your general enthusiasts that you would not let near your machine unless you think the idea of a monkey in your server room is somehow helpful (I would fall into the last category). Then again, I guess it would be quite the intruder detection system.

I remember very well (as do most) a time when those in the ecosystem would talk a lot about 'Open Source vs Enterprise'.

read more

I spend a lot of time kicking off EC2 instances for testing, bug reproduction, general information gathering, etc. These instances don’t have any sort of long-term life. Every so often I simply want to kill them all off, and I want to do so using the command line.

I’ve created ~/terminate-all-instances-in-region with permissions of 700.

$ cat terminate-all-instances-in-region
euca-describe-instances | grep INSTANCE | sed 's/INSTANCE[[:space:]]*\(i-[[:alnum:]]*\).*/\1/' | xargs euca-terminate-instances

Change euca- to ec2- if you are using the Amazon EC2 API Tools.

This assumes that your EC2 region is set via the EC2_URL environment variable. See this post for more details.

Filed under: Amazon

Let’s talk for a moment about configuring your Linux system to work with Amazon EC2 via the command line.

The command line tools that you’ll want are either the Amazon EC2 API Tools or the euca2ools package, depending on your language, license, and distro preferences.

In the examples below, change euca- to ec2- if you are using the Amazon EC2 API Tools.

Note that everything in this blog post is something that you should only have to do once, and then you’ll be off and running with EC2 for a long time to come.

Setting environment variables

The first thing to do is to configure your system’s environment variables to handle AWS account credentials. Create ~/set-ec2-environment as follows:

$ cat ~/set-ec2-environment
EC2_ACCESS_KEY=TEXT_FROM_WEBSITE_SEE_BELOW
EC2_SECRET_KEY=TEXT_FROM_WEBSITE_SEE_BELOW
EC2_CERT=~/amazon-cert-pub.pem
EC2_PRIVATE_KEY=~/amazon-cert-priv.pem
export EC2_ACCESS_KEY EC2_SECRET_KEY EC2_CERT EC2_PRIVATE_KEY

The values for these variables are all found or generated via this link. Some really useful docs are here.

Setting your region

EC2 is split into distinct regions. Typically you’ll choose a region based on your geographic location, and you will launch Amazon Machine Images (AMIs) in that region. For the most part, you should be able to do all of your work in one region, unless you make a conscious choice to spread your workload across regions, or if an AMI that you want to run is only available in a specific region.

$ euca-describe-regions
REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com

Create one or more ~/set-region-REGION-NAME as follows:

$ cat ~/set-region-us-east-1
EC2_URL=https://ec2.us-east-1.amazonaws.com:443
export EC2_URL

Tying credential and regional settings together

Edit ~/.bashrc to source the two configuration files on login, or just source the two files from the command line. Also, if you have multiple set-region-REGION-NAME files, it makes it very easy to change your region, simply by running source on the new region file.

source ~/set-ec2-environment
source ~/set-region-us-east-1

The reason why we’re going to all this trouble is because everything in EC2 is divided by regions, and the idea is to separate the global AWS configuration from the region currently in use, and to make it trivial to change that region from the command line.

Setting your EC2 ssh key

Now that you have your region set, it’s time to create your ssh key and upload it to the region to which your environment is pointing.

$ euca-add-keypair amazon-ssh > amazon-ssh
$ chmod 600 amazon-ssh


The default AWS security group in each region doesn’t allow inbound ssh access. It is a very simple command to enable this for all of your instances in that region.

$ euca-authorize -p 22 default

Finally, edit ~/.ssh/config to set the proper identify file for EC2 logins:

$ cat ~/.ssh/config
Host *.amazonaws.com
    User ec2-user
    IdentityFile ~/amazon-ssh

NOTE: It is possible to use a single SSH key for multiple regions, but euca2ools 1.3.1 doesn’t currently support this. You have to generate your own ssh public/private keypair, and then use ec2-import-keypair or the EC2 console in order to upload that public key to multiple regions.

Congrats! You’ve now finished all the one-time setup that is necessary to use EC2.

Launching your AMI

Launch your instance by running: $ euca-run-instances -k amazon-ssh AMI_ID

I have added alias euca-run-instances="euca-run-instances -k amazon-ssh" to my ~/.bashrc which allows me to simply run $ euca-run-instances AMI_ID with no additional command line arguments needed, unless I choose to specify a particular instance type, etc.

Connecting to your AMI

Run $ euca-describe-intances to get a list of all instances you have running in the region. You’ll see the hostname of the instance that you just started, and you can now run $ ssh HOSTNAME to connect. If everything is configured properly, you won’t need any other command line options.

Summary

Your home directory should contain:

amazon-cert-priv.pem
amazon-cert-pub.pem
amazon-ssh
set-ec2-environment
set-region-us-east-1

Your ~/.bashrc should contain:

alias euca-run-instances="euca-run-instances -k amazon-ssh"
source ~/set-ec2-environment
source ~/set-region-us-east-1

Your ~/.ssh/config should contain:

Host *.amazonaws.com
    User ec2-user
    IdentityFile ~/amazon-ssh

Filed under: Amazon

9.26.11 | CloudTweaks | Written by: Kiril Kirilov | Cisco Survey

The Internet Is Fundamental Resource for the Humankind, Cisco Survey

Future leaders, workers, and customers will rely increasingly on cloud resources

More than half of all students and young professionals consider Internet as an “integral part of their lives,” according to Cisco’s Connected World Technology Report 2011. In fact, the report’s findings are comparable and somewhat similar to the results published in 2010, when the first such report was conducted. The next generation of leaders and workers is so accustomed to a Internet-rich life that the years to come would witness growing number of connected devices and gadgets, more mobile lifestyle and booming cloud market to meet the fast growing expectations of young people not to rely on fixed storage devices for their data and software applications they use.

The Internet is now a fundamental resource for the humankind with 33 percent of those polled considering it is of equal importance to their daily life as air, water, food, and shelter, according to the survey. Almost half of the respondents, or 49 percent of college students and 47 percent of employees, younger than 30, believe the World Wide Web and the Internet are “pretty close” to the level of importance water, food and shelter have for the human race. Overall, four of every five college students and young professionals is of opinion the Internet is vital part of their daily life although it would be interesting to see a study, asking questions why and how the Internet is vital for young peoples’ lives.

The majority of young employees, 62 percent, and 55 percent of college students polled believe their life’s daily sustenance is in jeopardy if they are denied access to the Internet, while 64 percent of students would select an Internet connection instead of a car. A very interesting choice indeed, especially in the light of global environmental issues and air pollution that cars produce globally.

Online interaction is becoming integral part of the lifestyle of the next generation with 27 percent of college students preferring to update their social network profiles, while social networks are more important to them than partying, dating, listening to music, or hanging out with friends. On paper, it should be good news for online social networks that compete fiercely in the cloud but it is an alarming trend too for contacts in person cannot be substituted by online chats, psychologists agree.

The coming customers are going to be increasingly mobile after the study found that 66 percent of college students and 58 percent of young professionals consider their mobile devices like laptops, tablets, or smarthphones “the most important technology in their lives”. Therefore, we are to witness an unprecedented growth of mobile devices and applications market with cloud services destined to play an important role in this mobile revolution due to the overwhelming volume of data people are going to store, share, and use online.

Moreover, smartphones and desktops are now equally important to the next generation: 19 percent of students believe their smartphone is their “most important device” they use daily while 20 percent consider their desktop as their ultimate device. Thus, hardware and software vendors will be forced to switch to a new type of highly mobile customers that will ask for more power, applications, functionality, and productivity offered by their smartphones. Part of the solution is shift to cloud-based services but telephone makers will be forced to seek new hardware solutions as well.

In fact, corporations would be forced to change their business and Internet strategies, including cloud adoption and cloud services, much faster than expected, according to Marie Hattar, vice president, Enterprise Marketing, Cisco. “The results of the Cisco Connected World Technology Report should make businesses re-examine how they need to evolve in order to attract talent and shape their business models. Without a doubt, our world is changing to be much more Internet-focused, and becomes even more so with each new generation,” she said in a statement.

Click here to continue reading or visit www.cloudtweaks.com.

 Cisco’s Connected World Technology Report 2011

Today, Aaron Newcomb and Simon Phipps of FLOSS weekly, graciously hosted me on their podcast. They asked me about Eucalyptus, covering the basics (what is cloud computing, what is Eucalyptus) and then quickly going deeper into the internals of Eucalyptus, our development processes, and our community engagement. Very interesting ground was covered, ranging from cathedral vs. bazaar to the age old question of vi vs. emacs.

Check out the podcast. If you have questions about Eucalyptus that were not covered, you can always ask us directly.

9.23.11 | CloudTweaks | Written by: Charles Buck, COO and Co-Founder of independenceIT

Microsoft raised some eyebrows when Corporate VP Brad Anderson boldly proclaimed that, “Virtualizations is not cloud computing.” While this statement garnered a lot of buzz due to its timing – Anderson said this during the VMWorld annual event – that doesn’t make this statement any less true.

It is critical to remember that the principle concepts behind both cloud computing and virtualization have been in practice for decades. It is only enormous leaps in PC hardware architecture and the reliability of the Internet that have brought these terms into the general public’s accepted technical vocabulary. Virtualization and cloud computing are major catalysts to flexibility and innovation in the deployment of business applications. However, they have been erroneously lumped together by the public and I’d like to take a moment to set the record straight.

Virtualization is a systems admin and data management tool that has many technical uses; most of which have nothing to do with the cloud. The technology allows enterprises to use a single piece of physical hardware, to perform the work of many. Multiple operating system instances running on one hardware device are far more economical than a single piece of hardware for every server task. Still, there is no direct link to the cloud from pure virtualization.

Cloud computing, on the other hand, is access through the Internet to business applications running in a non-local environment. Cloud computing can certainly take advantage of virtualization but cloud computing can be (and has been) accomplished without the use of virtualization.

One way to describe the difference between virtualization and cloud computing in non technical terms is through the following thought experiment. Imagine you could be controlled by taking command inputs from another person – like you had a keyboard interface in your brain. Now imagine that the person controlling you is viewing you through a closed circuit TV link (a network), that is cloud computing.

To describe virtualization, imagine that you are standing between two mirrors, and now you see dozens of copies of yourself, only each copy was capable of doing tasks independent of the other copies. All the copies are “on” one piece of hardware (the real you). That is virtualization.

To bring the two together you would take all those virtualized copies of you and beam them out over many different links to an individual(s) who would control and interact with them remotely. Now, you have cloud computing leveraging virtualization!

The delineation between virtualization and cloud computing is important to make at this critical juncture in the technologies’ adoption.

Unlike most IT projects, cloud computing and virtualization impact almost everyone in the enterprise which means that non-technical people are involved in the implementation and deployment processes. By trying to “dumb down” the technical language, these vendors are leading their customers astray. This is not a simple case of semantics as there are distinct circumstances as to why why an organization would prefer virtualization over cloud computing and vice versa – especially since the two concepts are not two of the same thing.

Cloud computing and virtualization may be the lynch pin to modernizing today’s IT business environment. Together, they are ushering in a new era where companies are granted freedom to run their workspaces without having to conduct non-strategic IT functions such as patches, updates, and backups. The risk of not knowing the difference between virtualization and cloud computing can be a costly one as hype around these two technologies reaches a crescendo. Do yourself a favor and learn about the strengths and weaknesses of virtualization and cloud computing and map them to your organization’s needs before moving forward with an implementation.

 

Today is the release of the 2011.09 Amazon Linux AMI.

The AMI IDs are listed near the bottom of the detail page, along with the release notes.

If you are running the command line tools for accessing EC2, you can find the AMI IDs in your region by running:

$ euca-describe-images -o amazon | grep 2011.09.1 | grep amzn.*ami

Change euca- to ec2- if you are using the Amazon EC2 API Tools.

Filed under: Amazon

Some call it eat your own dog food but we prefer the alternative (check the wikipedia article if you don't know), and so we want to give you the latest experience in running our services into Eucalyptus, or, in other terms, how we have been drinking our own Champagne.

read more

Although summer is here, it's not vacation time for Eucalyptus. We have quite a few activities brewing. The next generation of Eucalyptus is getting ready to take your cloud to the next level, see our roadmap with the highlights of the new features) We'll post additional information soon with details about the status of the code, why it's not yet in our public repository, and why this secluded development won't happen again. We even have a video about it, but don't worry: there's no music or dancing engineers.

FastStart! While Eucalyptus got easier and easier to install, there is always room for improvement and, of course, David provided just the thing! He just finished FastStart, a set of scripts to simplify the deployment of Eucalyptus within a well-defined environment. It's great for small deployments!

More...

read more

The Eucalyptus team is pleased to announce the release of Eucalyptus 2.0.3. This update resolves the security issue identified in ESA-02: SOAP interfaces vulnerable to XML Signature Element Wrapping attacks.

This vulnerability allows an unauthenticated remote attacker who has access to the network traffic between authenticated user and a Eucalyptus installation, to modify intercepted SOAP requests and submit valid commands to the Eucalyptus SOAP interface. Special thanks to Juraj Somorovsky, Jörg Schwenk, Meiko Jensen and Xiaofeng Lou who alerted us to this vulnerability, thereby giving us all the needed details to produce the current release.

More...

read more

Eucalyptus has few components: the Cloud Controller, Walrus, the Storage Controller, the Cluster Controller and the Node Controller. In particular, the components our Eucalyptus user may be more familiar with, are the Cloud Controller and Walrus. The Cloud Controller implements the EC2 API and Walrus implements the S3 API. See our Developer's Corner for more information about Eucalyptus supported API.

Mark Atwood, our fearless Community Manager, ventured the trip half-way across the globe to linux.conf.au and while enjoying the conference, had a nice presentation about Eucalyptus and Walrus in particular. Watch Marks' presentation at http://blip.tv/file/4740872/.

More...

V2.0 of Eucalyptus Beginner’s Guide – UEC edition is out. This covers UEC on Ubuntu 10.10 Maverick Meerkat, based on Eucalyptus 2.0. Please download the PDF and post your comments.

To assist our community in maintaining secure installations we have created a Eucalyptus Security Advisory system for processing reported security issues. Community members can submit security questions and possible security issues to our security team. All reported issues will be reviewed promptly. Should a security issue be confirmed, it will receive a security rating (Critical, Moderate, or Low) and a Eucalyptus Security Advisory will be released containing information about the issue along with the recommended solution.

For more information, see our Eucalyptus Security Information page.

Contact our security team at security@eucalyptus.com.

More...

Due to some changes effected in the latest update of UEC packages on Ubuntu 10.04 Lucid Lynx, we have updated the Eucalyptus Beginner’s Guide to reflect those changes. Thus we have Eucalyptus Beginner’s Guide – Version 1.1. We’re also working on the Eucalyptus Beginner’s Guide on Ubuntu 10.10 Maverick Meerkat. Stay tuned to the blog to stay informed. Feedback is welcome.

Click the following link to download the book:

http://cssoss.files.wordpress.com/2010/11/eucalyptus-beginners-guide-uec-edition1-1.pdf

We have noticed a few changes in the way Eucalyptus 1.6 (UEC on Ubuntu 10.04 Lucid Lynx) and 2.0 (UEC on Ubuntu 10.10 Maverick Meerkat) launch instances. These changes have an impact on the bundling procedure. This article takes these changes into account and may be helpful for users of UEC on Ubuntu 10.10 and Eucalyptus 2.0

Eucalyptus Machine Image(EMI)

A Eucalpyptus Machine Image(EMI) is a combination of a virtual disk image(s), kernel and ramdisk images as well as an xml file containing meta data about the image. These images reside on WS3 and used as templates for creating instances on UEC. Each Linux EMI is a combination of the following:

• An XML file with a name like “jaunty.img.manifest.xml” with information about one or more hard disk images, a kernel image and a ram disk image (id – emi-65440E7E)
• An XML file with a name like “vmlinuz-2.6.28-11-server.manifest.xml” with information about the corresponding kernel image(id – eki-39FC1244)
• An XML file with a name like “initrd.img-2.6.28-11-server.manifest.xml” with information about the corresponding ramdisk image(id – eri-71ED1322)

Each of these images has its own ID that can be used while running the instances. More on this in the chapter on “Managing Instances”
From the web interface of Eucalyptus, you can view a list of EMIs in the “Store” tab of the web UI. These are the EMIs listed from Canonical or partners. You can choose to download and install any of these images directly from Canonical’s site.
Since most enterprise/individual users of Eucalyptus have a need for bringing up instances based on custom images, image management plays a key role in Eucalyptus administration. Such images could be based on a preferred version/variant of a preferred OS distribution with a set of required applications pre-installed.
Bundling an EMI is a multi-step process involving the following:

1. creating a virtual disk image
2. installing the OS
3. installing required applications
4. making the OS ready to run under UEC
5. registering the images with UEC
6. testing the image etc.

Bundling Linux Image

Client1 is machine used for the purpose of working on creation of the image. Please note that we had installed KVM on Client1.

Creating a new disk Image

This will represent the main HDD of the virtual machine, so make sure to give it as much space as you will need.

uecadmin@client1:~$ kvm-img create -f raw image.img 5G

Important note: The option “qcow2″ doesn’t work on UEC-Maverick Meerkat and Eucalyptus 2.0 installations. Hence it is important to create the KVM image in raw format, as eucalyptus starts the instance in raw option. If the image is created using qcow2 format,then while running the instance the console log of the instance gives an error saying “ALERT! /dev/sda1 does not exist. Dropping to a shell!”

OS Installation

Download the iso file of the Linux distribution you want installed in the image. The example below refers to creating a Maverick Meerkat 64-bit server image.

uecadmin@client1:~$ wget http://releases.ubuntu.com/10.10/ubuntu-10.10-server-amd64.iso 

And start the installation process:

uecadmin@client1:~$ sudo kvm -m 256 -cdrom ubuntu-10.10-server-amd64.iso -drive file=image.img,if=scsi,index=0 -boot d -net nic -net user -nographic -vnc :0

If your installation process requires more than 256MB of RAM change the -m option, and if you need more processors available, you can use the ‘-c’ option.The command above will boot a new KVM instance, with the disk image you’ve created as the primary HDD and the iso as the first bootable device. Also the ‘-nographic’ option will not display any graphical output. You can connect to the instance through VNC (use display number :0) and finish the installation.

For Ex: vncviewer A.B.C.D :0, where A.B.C.D is the IP address of Client1.

After finishing the installation, relaunch the VM by executing the following command.

uecadmin@client1:~$ sudo kvm -m 256 -drive file=image.img,if=scsi,index=0,boot=on -boot c -net nic -net user -nographic -vnc :0

At this point you can add all the packages you want to have installed, update the installation, add users and any settings that need to be present in your new UEC instances.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install mediawiki

Integrating with Eucalyptus

An instance running under Eucalyptus needs to know what IP it has and also, it needs to have the public key of the user allowed to do a passwordless access through SSH. This is accomplished by using a restful interface provided by the cloud. The interface is available under this URL: http://169.254.169.254/latest/meta-data and accessible from within the Instance.
Eucalyptus node controller is set up to prevent automatic key injection if the system is in MANAGED or MANAGED-NOVLAN mode. Instead, it is assumed that the instance will use the above meta-data service to retrieve the public keys when running in these modes. You will need to facilitate this by installing curl and adding a script that will run on each boot.
Install curl on the VM.

$ sudo apt-get install curl

Now add the following lines to /etc/rc.local of the image.

depmod -a
modprobe acpiphp
# simple attempt to get the user ssh key using the meta-data service
# assuming “user” is the username of an account that has been created
mkdir -p /home/user/.ssh
echo >> /home/user/.ssh/authorized_keys
curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /home/user/.ssh/authorized_keys
echo "AUTHORIZED_KEYS"
echo "************************"
cat /home/user/.ssh/authorized_keys
echo "************************"

Add the above lines before the “exit 0” in /etc/rc.local
Also remove the network persistent rules from /etc/udev/rules.d, so that the instance always comes up with eth0 as the interface name as expected by eucalyptus.

$ sudo rm -rf /etc/udev/rules.d/70-persistent-net.rules

This completes the process of customizing the OS installed as a reference image.

Registering with Eucalyptus

The last step would be to upload the images to Eucalyptus. The files that need to be uploaded for the above sample setup are: vmlinuz-2.6.28-11-server, initrd.img-2.6.28-11-server, image.img.
Copy the kernel and the initrd image from the VM image to some place outside. These will be used later for creating and uploading a complete virtual image to Eucalyptus.

$ scp /boot/initrd.img-2.6.35-22-server user@A.B.C.D:
$ scp /boot/vmlinuz-2.6.35-22-server user@A.B.C.D:

Before starting the upload process shut down the VM.

Registering kernel image

Execute the following commands to bundle and register the kernel image (vmlinuz-2.6.28-11-server)

uecadmin@client1:~$ euca-bundle-image -i vmlinuz-2.6.35-22-server --kernel true
uecadmin@client1:~$ euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.35-22-server.manifest.xmluecadmin@client1:~$ euca-register mybucket/vmlinuz-2.6.35-22-server.manifest.xml

Save the output produced by the last command above (eki-XXXXXXXX), which will be needed while registering the disk image.

Registering ramdisk image

Execute the following commands to bundle and register the ramdisk image (initrd.img-2.6.28-11-server)

uecadmin@client1:~$ euca-bundle-image -i  initrd.img-2.6.35-22-server
uecadmin@client1:~$ euca-upload-bundle -b mybucket -m /tmp/initrd.img-2.6.35-22-server.manifest.xml
uecadmin@client1:~$ euca-register mybucket/initrd.img-2.6.35-22-server.manifest.xml

Save the output produced by the last command above (eri-XXXXXXXX), which will be needed while registering the disk image.

Registering disk image

Execute the following commands to bundle and register the ramdisk image (image.img)

uecadmin@client1:~$ euca-bundle-image -i image.img --kernel eki-XXXXXXXX --ramdisk eri-XXXXXXXX
uecadmin@client1:~$ euca-upload-bundle -b mybucket -m /tmp/image.img.manifest.xml
uecadmin@client1:~$ euca-register mybucket/image.img.manifest.xml

Replace eki-XXXXXXXX and eri-XXXXXXXX with the exact values you have saved earlier.

Image Listing

The new images that have been uploaded can be seen by using euca-describe-images command.
For Ex:

uecadmin@client1:~$ euca-describe-images
IMAGE   emi-70B70EC0    mybucket/image.img.manifest.xml                     admin   available        public  x86_64  machine
IMAGE   eri-A2BE13EC    mybucket/initrd.img-2.6.35-22-server.manifest.xml   admin    available       public  x86_64  ramdisk
IMAGE   eki-685F1306    mybucket/vmlinuz-2.6.35-22-server.manifest.xml      admin    available       public  x86_64  kernel

More details on managing the instances in later chapters.

We have just released the PDF version of “Eucalyptus Beginner’s Guide – UEC Edition, V1.0″ and you can download it from http://cssoss.files.wordpress.com/2010/06/book_eucalyptus_beginners_guide_uec_edition1.pdf.  Please help us enhance the book by posting your suggestions as comments here.

The latest version of the pdf will also be availble from http://www.csscorp.com/enterprise-it-support/open-source-services.php .