Better Living Through Narratology

Adding aws-cfn-bootstrap (CloudFormation helper scripts) to custom Linux AMIs

As more and more of my CloudFormation (CF) stacks use a base image and CloudFormation::Init magic, it’s become imperative to have an AMI that has the helper scripts (cfn-signal, cfn-init, etc.) built-in.  This isn’t a problem if you use the Amazon Linux AMI, but if you’re playing with things like immutable infrastructure or baking your own custom AMIs for CIS hardening or some other regulatory requirement, it can become a big issue quickly.  There’s a little documentation out there on installing the CF helper scripts (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html) but the installation process is not quite so straightforward as one would hope.

The solution to this issue varies depending on your OS.  I’ve had no issue with Windows AMIs because the Ec2Config service takes care of everything, but in CentOS and RHEL, there are a few extra steps.  I’ll break them down by OS.  Note that you may need to search for updated version of things like epel-release to make sure it matches your OS or you’re using the most current version.

CentOS 6

This was relatively painless, thanks to the contents of the cloudformation-examples bucket being publicly visible.  The latest version of the helper-scripts requires some Python elements/versions that are a pain to set up, but you can use an older version of the helper scripts without any issues.  As of CentOS 6.8, you can (more…)

Posted in AWS, CloudFormation | No Comments


CloudFormation cfn-init unable to connect

I was having trouble getting a Windows CloudFormation stack that leveraged CloudFormation::Init (cfn-init) to work properly.  All I found in the cfn-init.log file was a repeating error that looked like this:

Traceback (most recent call last):
File “cfnbootstrap\util.pyc”, line 159, in _retry
File “cfnbootstrap\util.pyc”, line 231, in _timeout
ConnectionError: (‘Connection aborted.’, error(10060, ‘A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond’))
2016-03-24 20:13:40,355 [DEBUG] Sleeping for 0.873665 seconds before retrying

I ran through some google searches but came back with surprising few hits on this error.  Everything regarding the syntax of the stack was clean.  It passed all checks, came back as valid, etc..  But I got this error every time I deployed the stack.  The culprit?  Internet connectivity.  The stack referenced a file to download and was unable to do so thanks to some strict Network ACLs.  I had seen this issue once before when a NAT instance in the VPC was stopped.  The lesson: check your internet connectivity when dealing with this error.

Posted in AWS, CloudFormation | No Comments


AWS Lambda function to Stop, Snap and Start EC2 instances

AWS Lambda functions currently have a five minute time limit to execute and while this is not a big problem many functions, it becomes problematic when you’re executing a task that has some inherent latency.  I created a function that stops all instances, create snapshots of all attached EBS volumes and starts those instances back up.  This was easily feasible in my personal environment, but when you get to larger environments, the amount of time it takes to stop all instances – and back up all those volumes without hitting a CreateSnapshot limit – can easily exceed five minutes.

The solution is two-fold.

First, make sure you insert an increasing or variable sleep timer between creating snapshots.  I had to do this for the CreateSnapshot limit issue.

Second, in order to shut down all your instances properly, create snapshots of volumes and start instances back up, I had to use three separate functions and chain them together through the magic of CloudWatch and SNS.

Here’s how it works:

The function will output logs in CloudWatch.  When you find those logs, you’ll usually see something akin to “END RequestId” when the function has completed.  You can create a metric filter in that log group that looks for “END RequestId.”  Once that filter is created, you can create an alarm with it.  The alarm will trigger when the metric filter has been met and, if configured to do so, it can send a notification to an SNS topic of your choice.

The SNS topic can be tied to a Lambda function and should be considered a trigger to get the next function started.  Tie your CloudWatch alarm for the function that shuts down instances to the SNS topic that is tied to your backup function.  Go through the same process of creating a CloudWatch metric filter with an alarm and have that alarm notify a second SNS topic.

The second SNS topic should be tied to a Lambda function that will start your instances back up again.

So, in essence, we’re chaining everything together like this:

Lambda function > CloudWatch log > Metric Filter > Alarm > SNS topic > next Lambda function > CloudWatch log > Metric Filter …  and so on.  You can daisy chain these Lambda function together ad infinitum to meet your desired effect.

Posted in AWS, Lambda | No Comments


KMS, S3 and S3 Sync of folders instead of files

I recently tried to use AWS CLI to upload a folder full of files to S3 using a custom KMS key.  This is possible by using the “aws s3api put-object” command, but it’s not possible using the “aws s3 sync” command.  If you’re just uploading a few files, this isn’t a big deal, but the frustration grows with each extra file that needs to be uploaded.

The “s3 sync” command is a container for the s3api PUT action, so in order to use it for an entire folder (with a custom KMS key), you would need to write some kind of wrapper for it.

Otherwise, you can use one of the stock encryption keys and upload your entire folder to S3.

Posted in AWS, S3 | No Comments


Automating a Steam Gaming instance in AWS

A while ago, I went through the setup here ( http://lg.io/2015/07/05/revised-and-much-faster-run-your-own-highend-cloud-gaming-service-on-ec2.html ) to build a gaming machine in AWS and loved the result.  A lot cheaper to run one in the cloud than to shell out loads of cash for a new one and a great write-up.

The downside, although it was a one-time downside, was going through many of the settings to create the AMI that I needed and to tighten security a bit the way I needed.  Also, the article was written with a Mac client in mind and I run Windows.

So, with my Windows experience and with all the AWS work I’ve been doing lately, I put together a CloudFormation template to automate many of the steps.  If you’re looking (more…)

Posted in AWS, CloudFormation | No Comments


Moving Orphaned Secondary Site Systems in ConfigMgr 2007

Recently, I came across an issue in which a Secondary Site Server experienced unrecoverable hardware failure.  After the server was rebuilt, the old site was deleted with PreInst.exe /DELSITE and a new Secondary Site was created, I had to start re-adding all of the servers that were members of the failed Seconday Site.

For simplification, we’ll say the site code of the crashed Secondary Site is OLD and the site code of the newly created Secondary Site is NEW.  (Gotta love convenient three-letter words when it comes to SCCM!)

After adding the Site Systems and installing some component roles, I noticed ID 1048 in the Site Status error logs.  Error 1048 states: “SMS Site Component Manager detected that site system \\SERVERNAME is currently in use by SMS site OLD.”

OLD had been deleted but the Site Systems could not be removed before deletion, so they were, in effect, orphaned.  SCCM should  (more…)

Posted in IT, SCCM | Comments Off on Moving Orphaned Secondary Site Systems in ConfigMgr 2007


22 Basic Tips for Storytelling (according to Emma Coats)

This was shamelessly stolen from the internet but was too good not to add here, as well.

 

The 22 Things

#1: You admire a character for trying more than for their successes.

#2: You gotta keep in mind what’s interesting to you as an audience, not what’s fun to do as a writer. They can be very different.

#3: Trying for theme is important, but you won’t see what the story is actually about til you’re at the end of it. Now rewrite.

#4: Once upon a time there was ___. Every day, ___. One day ___. Because of that, ___. Because of that, ___. Until finally ___.

#5: Simplify. Focus. Combine characters. Hop over detours. You’ll feel like you’re losing valuable stuff but it sets you free.

#6: What is your character good at, comfortable with? Throw the polar (more…)

Posted in Writing | Comments Off on 22 Basic Tips for Storytelling (according to Emma Coats)


SCCM Keeps Processing Package

Occasionally, SCCM may seem to get stuck processing a package.  When viewing Status Messages, you’ll see :

SMS Distribution Manager successfully processed package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).
SMS Distribution Manager is beginning to process package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).
SMS Distribution Manager successfully processed package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).
SMS Distribution Manager is beginning to process package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).
SMS Distribution Manager successfully processed package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).
SMS Distribution Manager is beginning to process package “PRIN Windows 7 SP1 12.3″ (package ID = PRI00263).

This will keep going for sometime with no clear progress on distribution.  For example, if you’re trying to send a package to SEC-DP-SERVER and the package continues to be processed as above, then when you look at distmgr.log, you may see something like this:

Used 0 out of 3 allowed processing threads.
Sleep 3600 seconds…
Processing incoming file C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\distmgr.box\INCOMING\WHCXFL1G.PKG.
Adding package PRI00263 to the active package array.
StoredPkgVersion (1) of package PRI00263. StoredPkgVersion in database is 1.
Distribution Manager will not update the StoredPkgVersion in the Database as Despooler will.
Updated replicated package info for package PRI00263
No changed detected for package server [“Display=\\SEC-DP-SERVER1\”]MSWNET:[“SMS_SITE=SEC”]\\SEC-DP-SERVER1\ for package PRI00263, no need to update

Updated replicated package server [“Display=\\SEC-DP-SERVER\”]MSWNET:[“SMS_SITE=SEC”]\\SEC-DP-SERVER\ for package PRI00263

Updated replicated package program info for package PRI00263, program *, dependent program
Successfully processed C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\distmgr.box\INCOMING\WHCXFL1G.PKG, delete it
Removing package PRI00263 from the active package array.
Used 0 out of 3 allowed processing threads.
(more…)

Posted in IT, SCCM | Comments Off on SCCM Keeps Processing Package


Persist Task Sequence After Reboot

One of the nightmares projects I’ve been working on recently (with my guru co-worker ZenShaze) deals with integrating SCCM and a product that reverts workstations to a previous state.  Like a forced “snapshot” that will remove malware and unwanted programs with a single reboot.  The software essentially “anchors” specific files and directories and allows any changes made to those anchored targets to be saved after a reboot – everything else gets locked down and reverted to an earlier state.

Not a big issue when it comes to ConfigMgr running normally – until you run a Task Sequence that calls for a reboot.  When this happens, SCCM makes (more…)

Posted in IT, SCCM | Comments Off on Persist Task Sequence After Reboot


SCCM Report listing count of computers and associated hardware information

I needed a report today that would show me the amount of computers that we had of a certain make and model (easy enough) but the report also had to show the associated hardware information for those models.  Mildly problematic on a certain scale because someone with a specific model of computer out there is bound to have more or less RAM than someone else with the same model of computer.

After a good deal of fumbling around (SQL is not my strong suit), I pieced this together and it worked.

select
 GSCS.Manufacturer0 as [Manufacturer],
 GSCS.Model0 as [Model],
 MRY.TotalPhysicalMemory0 as [RAM],
 GSP.MaxClockSpeed0 as [CPU Speed (MHz)],
 GSLD.Size0 as [Disk Size (MB)],
 count(*) as [Count]
from v_R_System RS
left join v_GS_PROCESSOR GSP on RS.ResourceID = GSP.ResourceID
left join v_GS_X86_PC_MEMORY MRY on RS.ResourceID = MRY.ResourceID
left join v_GS_COMPUTER_SYSTEM GSCS on RS.ResourceID = GSCS.ResourceID
left join v_GS_LOGICAL_DISK GSLD on RS.ResourceID = GSLD.ResourceID
 Group by GSCS.Model0, MRY.TotalPhysicalMemory0, GSP.MaxClockSpeed0, GSCS.Manufacturer0, GSLD.Size0
 Order by GSCS.Model0

(more…)

Posted in IT, SCCM, SQL | Comments Off on SCCM Report listing count of computers and associated hardware information