What happened?

I purchased a 48 inch LED Samsung Smart TV UN48H6350AFXZA on November of 2014 right around the black Friday deals. I really wanted a smart TV as I was planning on only using it with online streaming services such as Netflix, Hulu Plus, and Amazon Prime Videos. I also have a local Plex Media Server that runs on my FreeBSD home server so I also wanted to be able to install the Plex app on my TV and stream it straight from the server without the need to use any additional device such as Chromecast.

The TV was working perfectly fine until a few weeks ago when I started getting occasional “Network error” when streaming movies using the Plex app. First I thought I had some issues with my Wireless router and didn’t pay much attention and restarted the TV. Things were fine but I kept getting the “Network error” message and the video will stop playing from time to time. I tried restarting the Wireless router and the Modem but no luck. I even tried resetting the TV to factory defaults. Nothing changed. After a few days, I couldn’t even connect to Wireless networks. I thought the Wi-Fi module in the TV stopped working and tried to use the Ethernet cable. Things were working fine for a few days and then it started acting up again. Finally, the TV would sometime freeze (i.e) not respond to any key strokes from the remove or the hardware jog control. Sometimes it will even reboot itself.

Samsung Cusomer Service

My TV came with just one year warranty and I didn’t purchase any extended warranty because I didn’t think I will need it but I was really disappointed when this problem happened. Since the TV was purchased over a year ago, I wasn’t sure if contacting the Customer Service was going to be any helpful but I contacted them anyway. I was on the chat with the customer services representative for hours and he asked me to try a lot of diagnostic steps which I had already done. They finally said the TV needs to be repaired and recommended a service center close to me.

So I called the service center and explained them all the symptoms and they said it would require changing the motherboard of the TV and that would cost about $400. I purchased the TV for only $600 and I wasn’t comfortable spending $400 just for getting it fixed.

Discover Extended Warranty

I made the purchase of the TV using my Discover card and one of the card benefits is Extended Product Warranty. So I decided to call Discover customer service to see if they would cover the cost for the repair. They got all the details from me regarding the TV and the problem, gave me a claim number and asked me to wait for a call from them.

The Adventure

I didn’t want to wait for them and tried to figure it out myself. These are the facts so far:

  • The TV started having trouble with Wi-Fi connectivity.
  • The TV failed to connect to Wi-Fi networks.
  • I use the Wi-Fi connection extensively.

I found out about the hidden service in the Samsung Smart TVs. It can be activated by the following steps:

  • Power the TV off.
  • Press Info + Menu + Mute + Power simultaneously.
  • Press the following keys in the order: Mute > 1 > 8 > 2 > Power

The TV will now power on and the service menu will show up. The service menu has a lot of options we can tweak. I didn’t want to touch anything that I wasn’t sure about. There is a menu to perform diagnostic tests on various components/module on the TV under the SVC option.

I did the diagnostic tests on the components one by one. When I tested the Wi-Fi module, the result came back as “Fail” and it also caused my TV to freeze again. I left it for a few seconds and then the TV rebooted. So this made it clear that the TV had trouble talking to the Wi-Fi module and that probably caused a Kernel panic which resulted in the reboot. The Samsung Smart TV runs linux kernel 2.6. Even though the TV initially started out with the Wi-Fi problem, because of all other issues I thought the Wi-Fi problem was just a symptom of something big but that wasn’t the case.

To confirm this, I took the TV apart and identified where the Wi-Fi module is located. Then I removed the connector that connects the Wi-Fi module to the motherboard. The problem went away. I couldn’t connect to any Wi-Fi networks but that makes sense. I was able to successfully use my Ethernet connection and stream movies just fine.

I looked up on eBay for the Wi-Fi module’s model number and part number and I was able to find one that costs me $8. I’ve ordered it and it should be here next week. I hope if I replace the Wi-Fi module, I can connect to Wi-Fi networks again.

Update (Jun 15th, 2016)

Today, I received the Wi-Fi module I ordered on eBay and replaced the faulty one I had on my TV and it has been working really well. But the new one only supports 2.4 GHz Wireless networks and not 5 GHz networks even though I bought the one with the same part number. It is not too big of a deal though.

Github notifications are pretty useful in a lot of cases but there are a couple of cases where there are no notifications sent. Examples for this are when someone watches your repository or someone forks it. Yes, these can be seen in your news feed if you watch the repository. But I have a lot of items in the news feed and it is easy to miss things. If I just created a project and want to see how people like it or interested in contributing to it, I would want to know when someone stars it or forks it so I can go check how they use the project or what contributions they want to make.

Initially I had a monitoring script running in one of my servers where it used the Github API to get the watchers and forks count and then sending me an email when someone watches or forks the repository I was interested in. This script was checking the repository status every 5 minutes and seemed unnecessary so I started looking for other options. Github has the option of setting up a webhook and you can specify exactly what events you want to get notified about. That sounded pretty interesting and I was playing with it a little bit. The services like travis-ci and Github pull request builder for Jenkins are using this webhook to get notified about the changes to the repository.

To create a new webhook,

Select particular events

The Github webhook makes a POST to the URL configured in the webhook with the information about what action took place in the repository. The following is the payload sent when someone watches the repository.

Request Headers
1
2
3
4
5
6
7
Request URL: http://192.0.2.10:4567/webhook
Request method: POST
content-type: "application/json"
Expect: ""
User-Agent: "GitHub Hookshot 5e2786c"
X-GitHub-Delivery: "54d2ff00-c501-11e3-9846-ff0dfba08f8b"
X-GitHub-Event: "watch"
JSON Payload
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
{
  "action": "started",
  "repository": {
    "id": 8544309,
    "name": "test_rubygem",
    "full_name": "arangamani/test_rubygem",
    "owner": {
      "login": "arangamani",
      "id": 1133812,
      "avatar_url": "https://avatars.githubusercontent.com/u/1133812?",
      "gravatar_id": "6ff021800637b88cbe17d1330cbcc1a5",
      "url": "https://api.github.com/users/arangamani",
      "html_url": "https://github.com/arangamani",
      "followers_url": "https://api.github.com/users/arangamani/followers",
      "following_url": "https://api.github.com/users/arangamani/following{/other_user}",
      "gists_url": "https://api.github.com/users/arangamani/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/arangamani/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/arangamani/subscriptions",
      "organizations_url": "https://api.github.com/users/arangamani/orgs",
      "repos_url": "https://api.github.com/users/arangamani/repos",
      "events_url": "https://api.github.com/users/arangamani/events{/privacy}",
      "received_events_url": "https://api.github.com/users/arangamani/received_events",
      "type": "User",
      "site_admin": false
    },
    "private": false,
    "html_url": "https://github.com/arangamani/test_rubygem",
    "description": "This is a simple test gem used for testing Rubygems API Client. Do NOT use.",
    "fork": false,
    "url": "https://api.github.com/repos/arangamani/test_rubygem",
    "forks_url": "https://api.github.com/repos/arangamani/test_rubygem/forks",
    "keys_url": "https://api.github.com/repos/arangamani/test_rubygem/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/arangamani/test_rubygem/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/arangamani/test_rubygem/teams",
    "hooks_url": "https://api.github.com/repos/arangamani/test_rubygem/hooks",
    "issue_events_url": "https://api.github.com/repos/arangamani/test_rubygem/issues/events{/number}",
    "events_url": "https://api.github.com/repos/arangamani/test_rubygem/events",
    "assignees_url": "https://api.github.com/repos/arangamani/test_rubygem/assignees{/user}",
    "branches_url": "https://api.github.com/repos/arangamani/test_rubygem/branches{/branch}",
    "tags_url": "https://api.github.com/repos/arangamani/test_rubygem/tags",
    "blobs_url": "https://api.github.com/repos/arangamani/test_rubygem/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/arangamani/test_rubygem/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/arangamani/test_rubygem/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/arangamani/test_rubygem/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/arangamani/test_rubygem/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/arangamani/test_rubygem/languages",
    "stargazers_url": "https://api.github.com/repos/arangamani/test_rubygem/stargazers",
    "contributors_url": "https://api.github.com/repos/arangamani/test_rubygem/contributors",
    "subscribers_url": "https://api.github.com/repos/arangamani/test_rubygem/subscribers",
    "subscription_url": "https://api.github.com/repos/arangamani/test_rubygem/subscription",
    "commits_url": "https://api.github.com/repos/arangamani/test_rubygem/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/arangamani/test_rubygem/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/arangamani/test_rubygem/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/arangamani/test_rubygem/issues/comments/{number}",
    "contents_url": "https://api.github.com/repos/arangamani/test_rubygem/contents/{+path}",
    "compare_url": "https://api.github.com/repos/arangamani/test_rubygem/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/arangamani/test_rubygem/merges",
    "archive_url": "https://api.github.com/repos/arangamani/test_rubygem/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/arangamani/test_rubygem/downloads",
    "issues_url": "https://api.github.com/repos/arangamani/test_rubygem/issues{/number}",
    "pulls_url": "https://api.github.com/repos/arangamani/test_rubygem/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/arangamani/test_rubygem/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/arangamani/test_rubygem/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/arangamani/test_rubygem/labels{/name}",
    "releases_url": "https://api.github.com/repos/arangamani/test_rubygem/releases{/id}",
    "created_at": "2013-03-03T23:17:23Z",
    "updated_at": "2014-04-15T05:04:11Z",
    "pushed_at": "2014-04-15T05:04:11Z",
    "git_url": "git://github.com/arangamani/test_rubygem.git",
    "ssh_url": "git@github.com:arangamani/test_rubygem.git",
    "clone_url": "https://github.com/arangamani/test_rubygem.git",
    "svn_url": "https://github.com/arangamani/test_rubygem",
    "homepage": "https://rubygems.org/gems/test_rubygem",
    "size": 216,
    "stargazers_count": 1,
    "watchers_count": 1,
    "language": "Ruby",
    "has_issues": true,
    "has_downloads": true,
    "has_wiki": true,
    "forks_count": 0,
    "mirror_url": null,
    "open_issues_count": 1,
    "forks": 0,
    "open_issues": 1,
    "watchers": 1,
    "default_branch": "master"
  },
  "sender": {
    "login": "arangamani",
    "id": 1133812,
    "avatar_url": "https://avatars.githubusercontent.com/u/1133812?",
    "gravatar_id": "6ff021800637b88cbe17d1330cbcc1a5",
    "url": "https://api.github.com/users/arangamani",
    "html_url": "https://github.com/arangamani",
    "followers_url": "https://api.github.com/users/arangamani/followers",
    "following_url": "https://api.github.com/users/arangamani/following{/other_user}",
    "gists_url": "https://api.github.com/users/arangamani/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/arangamani/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/arangamani/subscriptions",
    "organizations_url": "https://api.github.com/users/arangamani/orgs",
    "repos_url": "https://api.github.com/users/arangamani/repos",
    "events_url": "https://api.github.com/users/arangamani/events{/privacy}",
    "received_events_url": "https://api.github.com/users/arangamani/received_events",
    "type": "User",
    "site_admin": false
  }
}

Here is the simple ruby code written using Sinatra.

The Sinatra App
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
require 'sinatra'
require 'json'
require 'mail'

configure do
  set :bind, '192.0.2.10'
end

post '/webhook' do
  event = request.env['HTTP_X_GITHUB_EVENT']
  case event
  when 'watch'
    payload = JSON.parse(request.body.read)
    repo = payload['repository']['full_name']
    user = payload['sender']['login']
    user_url = payload['sender']['html_url']
    puts "This is a watch event on '#{repo}' by '#{user}': #{user_url}"
    mail = Mail.new do
      from "noreply@example.com"
      to "me@arangamani.net"
      subject "Repo #{repo} was starred by #{user}"
      body "Reposiroty: #{repo}\n" +
        "User: #{user}\n" +
        "User URL: #{user_url}\n" +
        "Total Stars: #{payload['repository']['stargazers_count']}\n"
    end
    mail.deliver!
  when 'fork'
    payload = JSON.parse(request.body.read)
    repo = payload['repository']['full_name']
    forkee_name = payload['forkee']['owner']['login']
    forkee_type = payload['forkee']['owner']['type']
    forked_repo = payload['forkee']['full_name']
    forkee_url = payload['forkee']['owner']['html_url']
    fork_url = payload['forkee']['html_url']
    puts "This is a fork event on '#{repo}' by '#{forkee_name}' (#{forkee_type})."
    puts "The fork '#{forked_repo}' is avaialble at #{fork_url}"
    mail = Mail.new do
      from "noreply@example.com"
      to "user@example.com"
      subject "Repo #{repo} was forked by #{forkee_name}"
      body "Reposiroty: #{repo}\n" +
        "Forked by: #{forkee_name} (#{forkee_type})\n" +
        "Forked repo #{forked_repo} available at: #{fork_url}\n" +
        "Forkee URL: #{forkee_url}\n" +
        "Total Forks: #{payload['repository']['forks_count']}\n"
    end
    mail.deliver!
  else
    puts "Received '#{event}' type event. Don't know what to do."
  end
  # This block is not required -- just for fun
  content_type 'application/json'
  JSON.pretty_generate(
    {
      message: {
        type: 'appreciation',
        body: 'Hey Github! Thanks for letting me know'
      }
    }
  )
end

Overview

Nobody likes when something fails. When a Chef run fails for some reason, we have to figure out what resource caused the failure and then try to fix it and then re-run it. If we know what might fail, we can try to do some steps and then try the action again. This message sent to the Chef community mailing list proposes something very similar.

Here is an RFC proposed by the Chef community which is still in discussion. It provides a resource specific error handling mechanism. It has many features that will help making an awesome overall Chef experience. If we are aware of something that can fail, why not make the resource try something to fix it by itself?

The Chef Handler provides a way of handling exceptions but that is very limited to just handling exceptions and performing some actions after the failure and be done. I wrote a blog post earlier about how to write custom Chef Handlers.

Proof of Concept

I wrote a simple cookbook called on_failure that monkey patches the way Chef runs the resources and implements this feature. I tried to add as many features as possible but there are still some discussions on how to do certain things. This is just a starting point for trying out how this would work. This cookbook is also published to the community.

In Action

The on_failure cookbook has some additional cookbooks that help trying out this feature. The following are those cookbooks and their purposes:

  • sample – Contains some sample recipes for using the on_failure handler feature.
  • meal – Provides a resource called meal to mimic the example provided in the RFC.
  • food – Provides a resource called food to mimic the example provided in the RFC.

I tried to make all the examples used here to be very similar to the ones proposed in the RFC to make them easier to understand and those examples are also fun to use.

Simple Example

Here is a simple example of how adding this feature to a resource will look like:

sample::default
1
2
3
4
5
6
7
meal 'breakfast' do
  on_failure { notify :eat, 'food[bacon]' }
end

food 'bacon' do
  action :nothing
end

This recipe doesn’t actually raise any error so there is no use for the on_failure handler in the meal[breakfast] resource. Let’s simulate a failure and see what the (sample::with_exception) recipe looks like:

sample::with_exception
1
2
3
4
5
6
7
8
9
10
# At least 1 bacon slice is required to complete breakfast without getting hungry.
node.override['meal']['bacon_required'] = 1

meal 'breakfast' do
  on_failure(HungryError) { notify :eat, 'food[bacon]' }
end

food 'bacon' do
  action :nothing
end

This recipe will raise a HungryError exception when meal[breakfast] resource first runs. During failure, the food[bacon] resource’s :eat action will be run and it will make the resource meal[breakfast] to succeed when it is tried again. See the output in the following Chef run:

Chef run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[2014-04-09T04:07:10+00:00] INFO: *** Chef 11.12.0 ***
[2014-04-09T04:07:10+00:00] INFO: Chef-client pid: 8683
[2014-04-09T04:07:12+00:00] INFO: Setting the run_list to ["recipe[sample::with_exception]"] from CLI options
[2014-04-09T04:07:12+00:00] INFO: Run List is [recipe[sample::with_exception]]
[2014-04-09T04:07:12+00:00] INFO: Run List expands to [sample::with_exception]
[2014-04-09T04:07:12+00:00] INFO: Starting Chef Run for on-failure-berkshelf
[2014-04-09T04:07:12+00:00] INFO: Running start handlers
[2014-04-09T04:07:12+00:00] INFO: Start handlers complete.
[2014-04-09T04:07:12+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 1 bacon slices. But I only ate: 0>
[2014-04-09T04:07:12+00:00] INFO: Ate 'bacon' successfully. yummy!
[2014-04-09T04:07:12+00:00] INFO: Meal 'breakfast' completed successfully.
[2014-04-09T04:07:12+00:00] INFO: Chef Run complete in 0.091023597 seconds
[2014-04-09T04:07:12+00:00] INFO: Running report handlers
[2014-04-09T04:07:12+00:00] INFO: Report handlers complete

Handling with Retries

By default, only one attempt is made to rerun the resource action. It can be customized by giving the retries option in the on_failure construct. The resource action will be tried for the specified number of times and the block given will be executed every time before retrying the resource action.

sample::with_retries
1
2
3
4
5
6
7
8
9
10
# 3 bacon slices are required for completing the breakfast without getting hungry. This is used to do retries.
node.override['meal']['bacon_required'] = 3

meal 'breakfast' do
  on_failure(retries: 5) { notify :eat, 'food[bacon]' }
end

food 'bacon' do
  action :nothing
end

In this example, the meal[breakfast] resource will succeed only if 3 bacon slices are eaten — so it will pass at the 4th attempt. See the output in the following Chef run:

Chef run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[2014-04-09T05:12:19+00:00] INFO: *** Chef 11.12.0 ***
[2014-04-09T05:12:19+00:00] INFO: Chef-client pid: 8919
[2014-04-09T05:12:21+00:00] INFO: Setting the run_list to ["recipe[sample::with_retries]"] from CLI options
[2014-04-09T05:12:21+00:00] INFO: Run List is [recipe[sample::with_retries]]
[2014-04-09T05:12:21+00:00] INFO: Run List expands to [sample::with_retries]
[2014-04-09T05:12:21+00:00] INFO: Starting Chef Run for on-failure-berkshelf
[2014-04-09T05:12:21+00:00] INFO: Running start handlers
[2014-04-09T05:12:21+00:00] INFO: Start handlers complete.
[2014-04-09T05:12:21+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 3 bacon slices. But I only ate: 0>
[2014-04-09T05:12:21+00:00] INFO: Ate 'bacon' successfully. yummy!
[2014-04-09T05:12:21+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 3 bacon slices. But I only ate: 1>
[2014-04-09T05:12:21+00:00] INFO: Ate 'bacon' successfully. yummy!
[2014-04-09T05:12:21+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 3 bacon slices. But I only ate: 2>
[2014-04-09T05:12:21+00:00] INFO: Ate 'bacon' successfully. yummy!
[2014-04-09T05:12:21+00:00] INFO: Meal 'breakfast' completed successfully.
[2014-04-09T05:12:21+00:00] INFO: Chef Run complete in 0.099481932 seconds
[2014-04-09T05:12:21+00:00] INFO: Running report handlers
[2014-04-09T05:12:21+00:00] INFO: Report handlers complete

Handling Multiple Exceptions

Multiple exceptions can be specified in on_failure and it will be caught if any of the exceptions are raised by the resource.

sample::multiple_exceptions
1
2
3
4
5
6
7
8
9
10
# Simlate that the meal is not cooked so UncookedError is raised
node.set['meal']['cooked'] = false

meal 'breakfast' do
  on_failure(UncookedError, HungryError, retries: 3) { notify :fry, 'food[bacon]' }
end

food 'bacon' do
  action :nothing
end

See the output in the following Chef run:

Chef run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[2014-04-09T05:14:08+00:00] INFO: *** Chef 11.12.0 ***
[2014-04-09T05:14:08+00:00] INFO: Chef-client pid: 9141
[2014-04-09T05:14:10+00:00] INFO: Setting the run_list to ["recipe[sample::multiple_exceptions]"] from CLI options
[2014-04-09T05:14:10+00:00] INFO: Run List is [recipe[sample::multiple_exceptions]]
[2014-04-09T05:14:10+00:00] INFO: Run List expands to [sample::multiple_exceptions]
[2014-04-09T05:14:10+00:00] INFO: Starting Chef Run for on-failure-berkshelf
[2014-04-09T05:14:10+00:00] INFO: Running start handlers
[2014-04-09T05:14:10+00:00] INFO: Start handlers complete.
[2014-04-09T05:14:10+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::UncookedError: Meal is not cooked well>
[2014-04-09T05:14:10+00:00] INFO: Meal 'bacon' is cooked
[2014-04-09T05:14:11+00:00] INFO: Meal 'breakfast' completed successfully.
[2014-04-09T05:14:11+00:00] INFO: Chef Run complete in 0.089208967 seconds
[2014-04-09T05:14:11+00:00] INFO: Running report handlers
[2014-04-09T05:14:11+00:00] INFO: Report handlers complete

Using Multiple on_failure Blocks

We may not want to do the same thing for all sorts of failures, so you can specify different on_failure blocks for different exception types. The blocks will be evaluated in the order they appear in the resource declaration.

sample::multiple_blocks
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Simulate that meal is not cooked so UncookedError is raised
node.set['meal']['cooked'] = false

# Simulate that meal is cold so ColdError is raised
node.set['meal']['cold'] = true

meal 'breakfast' do
  on_failure(UncookedError) { notify :fry, 'food[bacon]' }
  on_failure(ColdError) { notify :microwave, 'food[bacon]' }
end

food 'bacon' do
  action :nothing
end

See the output of the following Chef run:

Chef run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[2014-04-09T05:15:33+00:00] INFO: *** Chef 11.12.0 ***
[2014-04-09T05:15:33+00:00] INFO: Chef-client pid: 9362
[2014-04-09T05:15:35+00:00] INFO: Setting the run_list to ["recipe[sample::multiple_blocks]"] from CLI options
[2014-04-09T05:15:35+00:00] INFO: Run List is [recipe[sample::multiple_blocks]]
[2014-04-09T05:15:35+00:00] INFO: Run List expands to [sample::multiple_blocks]
[2014-04-09T05:15:35+00:00] INFO: Starting Chef Run for on-failure-berkshelf
[2014-04-09T05:15:35+00:00] INFO: Running start handlers
[2014-04-09T05:15:35+00:00] INFO: Start handlers complete.
[2014-04-09T05:15:35+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::UncookedError: Meal is not cooked well>
[2014-04-09T05:15:35+00:00] INFO: Meal 'bacon' is cooked
[2014-04-09T05:15:35+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::ColdError: Meal is cold>
[2014-04-09T05:15:35+00:00] INFO: Meal 'bacon' is now hot!
[2014-04-09T05:15:35+00:00] INFO: Meal 'breakfast' completed successfully.
[2014-04-09T05:15:35+00:00] INFO: Chef Run complete in 0.090865808 seconds
[2014-04-09T05:15:35+00:00] INFO: Running report handlers
[2014-04-09T05:15:35+00:00] INFO: Report handlers complete

It is clear that initially the Chef run failed because the food was not cooked. So it tried to fry the bacon so the food was then cooked but it was cold (what?? just for example) so it tried to microwave the food and hence the Chef run succeeded.

Access to Resource Attributes

The block specified in on_failure can take the resource as an argument so it can access all the resource’s attributes.

sample::resource_attributes
1
2
3
4
5
6
7
8
# At least 1 bacon slice is required to complete breakfast without getting hungry.
node.override['meal']['bacon_required'] = 1

# The handler block is just used to demonstrate that the resource attributes can be accessed.
meal 'breakfast' do
  time '2014-04-09 08:00:00 -0700'
  on_failure { |breakfast| Chef::Log.info "Tried eating breakfast at: #{breakfast.time}" }
end

See the output of the following Chef run:

Chef run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[2014-04-09T06:09:05+00:00] INFO: *** Chef 11.12.0 ***
[2014-04-09T06:09:05+00:00] INFO: Chef-client pid: 2919
[2014-04-09T06:09:07+00:00] INFO: Setting the run_list to ["recipe[sample::resource_attributes]"] from CLI options
[2014-04-09T06:09:07+00:00] INFO: Run List is [recipe[sample::resource_attributes]]
[2014-04-09T06:09:07+00:00] INFO: Run List expands to [sample::resource_attributes]
[2014-04-09T06:09:07+00:00] INFO: Starting Chef Run for on-failure-berkshelf
[2014-04-09T06:09:07+00:00] INFO: Running start handlers
[2014-04-09T06:09:07+00:00] INFO: Start handlers complete.
[2014-04-09T06:09:07+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 1 bacon slices. But I only ate: 0>
[2014-04-09T06:09:07+00:00] INFO: Tried eating breakfast at: 2014-04-09 08:00:00 -0700
[2014-04-09T06:09:07+00:00] INFO: meal[breakfast] failed with: #<MealExceptions::HungryError: I want 1 bacon slices. But I only ate: 0>

================================================================================
Error executing action `eat` on resource 'meal[breakfast]'
================================================================================


MealExceptions::HungryError
---------------------------
I want 1 bacon slices. But I only ate: 0
...

After the failure occurred, the block given in on_failure got executed which just had a single log line to show that it had access to the resource’s attributes.

Overview

Chef Handler is used to identify situations that may arise during a chef-client run and then instruct the chef-client on how to handle these situations. There are three type of handlers in Chef:

  • exception – Exception handler is used to identify situations that have caused the chef-client run to fail. This type of handler can be used to send email notifications about the failure or can take necessary actions to prevent from cascading failure.
  • report – Report handler is used when a chef-client run succeeds and reports back on certain details about that chef-client run.
  • start – Start handler is used to run events at the beginning of the chef-client run.

In this blog, we will see how to write a custom exception handler to avoid the failure of a chef run to cause big side effects.

Background

When I was working on the rs-storage cookbook, there was a situation where we freeze the filesystem during a backup. I wanted to make sure the filesystem freeze doesn’t give any adverse effects for the applications that may use the filesystem. The sequence of actions we do for taking a backup are: freeze the filesystem, take a backup using the RightScale API, and then unfreeze the filesystem. What if the backup operation fails? Will it leave the filesystem in a frozen state? Definitely yes. We don’t want that to happen. So the filesystem should be unfrozen regardless of whether the backup succeeded or failed.

Solution

To avoid leaving the filesystem from being frozen on a backup failure, I decided to write an exception handler which will unfreeze the filesystem during a backup failure.

The recipe looks like this

Original Recipe
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Attributes used by the resources
node.set['rs-storage']['device']['nickname'] = 'data_storage'
node.set['rs-storage']['device']['mount_point'] = '/mnt/storage'
node.set['rs-storage']['backup']['lineage'] = 'testing'

# Freeze the filesystem
filesystem node['rs-storage']['device']['nickname'] do
  mount node['rs-storage']['device']['mount_point']
  action :freeze
end

# Take a backup
rightscale_backup node['rs-storage']['device']['nickname'] do
  lineage node['rs-storage']['backup']['lineage']
  action :create
end

# Unfreeze the filesystem
filesystem node['rs-storage']['device']['nickname'] do
  mount node['rs-storage']['device']['mount_point']
  action :unfreeze
end

The filesystem cookbook was modified to support the freeze and unfreeze actions. The rightscale_backup provides a resource/provider for handling backups on a RightScale supported cloud.

This recipe is just that simple. But if the backup resource encountered an error, the chef run will and leave the filesystem in frozen state. So I wrote a small exception handler that gets run when the chef run fails and will notify the unfreeze action of the filesystem[data_storage] resource. Here is the handler:

The Error Handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
module Rightscale
  class BackupErrorHandler < Chef::Handler
    # This method gets called when the chef run fails and this handler is registered
    # to act as an exception handler.
    def report
      # The nickname (data_storage in this example) is obtained from the node
      nickname = run_context.node['rs-storage']['device']['nickname']
      # Find the filesystem resource from resource collection
      filesystem_resource = run_context.resource_collection.lookup("filesystem[#{nickname}]")
      # Run the `unfreeze` action on the filesystem resource found
      filesystem_resource.run_action(:unfreeze) if filesystem_resource
    end
  end
end

This handler will simply call the unfreeze action on the filesystem resource when the chef run fails. Place this handler in the files directory of your cookbook and we will use this later as cookbook_file. To enable this handler we will use the chef_handler cookbook.

Chef Handler Cookbook

The chef_handler cookbook published by Chef enables the creation and configuration of chef handlers easy. This cookbook creates the handlers directory and provides an LWRP for enabling and disabling specific handlers from within the recipe.

Here is the modified recipe with the use of chef_handler

Modified Recipe with Chef Handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Include the chef_handler::default recipe which will setup the handlers directory for chef.
include_recipe 'chef_handler::default'

# Place the handler file we wrote in the handlers directory on the node
cookbook_file "#{node['chef_handler']['handler_path']}/rs-storage_backup.rb" do
  source 'backup_error_handler.rb'
  action :create
end

# Enable the exception handler for this recipe
chef_handler 'Rightscale::BackupErrorHandler' do
  source "#{node['chef_handler']['handler_path']}/rs-storage_backup.rb"
  action :enable
end

# Attributes used by the resources
node.set['rs-storage']['device']['nickname'] = 'data_storage'
node.set['rs-storage']['device']['mount_point'] = '/mnt/storage'
node.set['rs-storage']['backup']['lineage'] = 'testing'

# Freeze the filesystem
filesystem node['rs-storage']['device']['nickname'] do
  mount node['rs-storage']['device']['mount_point']
  action :freeze
end

# Take a backup
rightscale_backup node['rs-storage']['device']['nickname'] do
  lineage node['rs-storage']['backup']['lineage']
  action :create
end

# Unfreeze the filesystem
filesystem node['rs-storage']['device']['nickname'] do
  mount node['rs-storage']['device']['mount_point']
  action :unfreeze
end

Now if the chef run fails, the filesystem will always be unfrozen.

Further Reading

There is an RFC proposed for chef called on_failure which gives a nice way for handling such exceptions. These on_failure blocks can be given at the resource level instead of the recipe level which is much nicer.

The Problem

It all started when I was editing a Vagrantfile and making more customization. It is simply ruby and having syntax highlighting in it will be easier to edit the file. I always see the following line in the Vagrantfile:

1
# vi: set ft=ruby :

but it didn’t seem to do anything for the syntax highlighting. I tried to set the filetype for the Vagrantfile when I was in the editor using

1
:set filetype=ruby

it worked like a charm. But I don’t want to do it every time I enter the file. Also I have the same problem with the Gemfile as well as the Berksfile.

The Solution

I did some research and figured out that I can achieve this using the autocmd in vimrc. So added the following lines to my vimrc.

1
2
3
autocmd BufNewFile,BufRead Gemfile set filetype=ruby
autocmd BufNewFile,BufRead Vagrantfile set filetype=ruby
autocmd BufNewFile,BufRead Berksfile set filetype=ruby

So whenever I open any of these files, they are highlighted as ruby files and I am happy!

Chef::Taste – What is it?

Chef Taste is a simple command line utility to check a cookbook’s dependency status. It will list the dependent cookbooks in a tabular format with the version information, status, and the changelog (if possible) for out-of-date cookbooks.

Inspiration

The Gemnasium project does a similar job for checking a Ruby project’s dependencies and keep them up-to-date. Chef-Taste is similar to that instead it provides a simple command line utility to list the dependency status for Chef cookbooks.

What it does?

When you are inside the cookbooks directory, simply type taste to taste the cookbook. The metadata.rb of the cookbook is parsed to obtain the dependencies. It will display a table that contains the following rows:

  • Name – The name of the cookbook
  • Requirement – The version requirement specified in the metadata
  • Used – The final version used based on the requirement constraint
  • Latest – The latest version available in the community site
  • Status – The status of the cookbook: up-to-date (a green tick mark) or out-of-date (a red x mark)
  • Changelog – The changelog of out-of-date cookbooks if available.

The overall status will also be displayed in the bottom of the table.

Changelog

Most of the cookbooks are hosted in Github and are tagged for every release. The changelog is computed by obtaining the source URL provided in the community site and finding the tags being used and the latest tag and displaying a compare view that compares these two tags. This URL is then shortened using goo.gl URL shortener to fit the table.

The details are obtained only for cookbooks available in the community site. Other cookbooks are displayed but will simply have N/A in their details.

Examples

These examples are based on the cookbooks available in the test/cookbooks directory in this repository.

1. fried_rice cookbook

1
2
3
4
5
6
7
8
9
10
kannanmanickam@mac fried_rice$ taste
+------------+-------------+--------+--------+--------+----------------------+
| Name       | Requirement | Used   | Latest | Status | Changelog            |
+------------+-------------+--------+--------+--------+----------------------+
| ntp        | ~> 1.4.0    | 1.4.0  | 1.5.0  |   ✖    | http://goo.gl/qsfgwA |
| swap       | = 0.3.5     | 0.3.5  | 0.3.6  |   ✖    | http://goo.gl/vZtUQJ |
| windows    | >= 0.0.0    | 1.11.0 | 1.11.0 |   ✔    |                      |
| awesome_cb | >= 0.0.0    | N/A    | N/A    |  N/A   |                      |
+------------+-------------+--------+--------+--------+----------------------+
Status: out-of-date ( ✖ )

2. noodles cookbook

1
2
3
4
5
6
7
8
9
kannanmanickam@mac noodles$ taste
+---------+-------------+--------+--------+--------+----------------------+
| Name    | Requirement | Used   | Latest | Status | Changelog            |
+---------+-------------+--------+--------+--------+----------------------+
| mysql   | ~> 3.0.12   | 3.0.12 | 3.0.12 |   ✔    |                      |
| apache2 | ~> 1.7.0    | 1.7.0  | 1.8.4  |   ✖    | http://goo.gl/9ejcpi |
| windows | >= 0.0.0    | 1.11.0 | 1.11.0 |   ✔    |                      |
+---------+-------------+--------+--------+--------+----------------------+
Status: out-of-date ( ✖ )

3. curry cookbook

1
2
3
4
5
6
7
8
9
10
kannanmanickam@mac curry$ taste
+-----------------+-------------+-------+--------+--------+-----------+
| Name            | Requirement | Used  | Latest | Status | Changelog |
+-----------------+-------------+-------+--------+--------+-----------+
| ntp             | >= 0.0.0    | 1.5.0 | 1.5.0  |   ✔    |           |
| lvm             | >= 0.0.0    | 1.0.0 | 1.0.0  |   ✔    |           |
| application     | >= 0.0.0    | 4.1.0 | 4.1.0  |   ✔    |           |
| application_php | >= 0.0.0    | 2.0.0 | 2.0.0  |   ✔    |           |
+-----------------+-------------+-------+--------+--------+-----------+
Status: up-to-date ( ✔ )

4. water cookbook

1
2
kannanmanickam@mac water$ taste
No dependent cookbooks

Problem

I use Vagrant for setting up my local development environments. Recently, I setup an OpenStack development environment on a vagrant virtual machine using the Devstack project. I brought up a Ubuntu 12.04 virtual machine using vagrant and checked out the Devstack code and started stacking. In the initial attempt everything worked fine and I had a working OpenStack devlopment environment with the up-to-date code from the master branch of all projects. The source code for all OpenStack projects will be checked out to /opt/stack directory on the virtual machine. You can simply modify the code on the virtual machine and restart the appropriate services to see the code change getting reflected. Editing the code inside a virtual machine using a terminal-based editor can be easier for simple code changes. But for complex code modification and writing new features, it will good to have a Python IDE. So I decided to use the synced folder feature available in vagrant. This feature allows you to share a folder from the host machine to the virtual machine as an NFS share. So the files from the host machine can be edited directly using an IDE and the changes can be seen in the virtual machine. Seems pretty simple, right? That’s what I thought and created a synced folder on my host machine will be mapped to the /opt/stack directory on the virtual machine. After a few weeks, I set this configuration on my Vagrantfile and reloaded my virtual machine. Everything went fine but when it tried to install and configure horizon, it just stuck forever. I thought it was a network issue and restarted the stack.sh a few times and tried to leave it running overngiht but no luck. Then I terminated the VM and started it on a fresh virtual machine. At this point, I didn’t know that the problem was with the synced folder. This issue was going on for a few weeks and I didn’t take the time to dig deep into the issue to find out the root cause of the problem. While casually trying to setup the Devstack on another laptop with same configuration and without the synced folder feature, it worked fine.

Research/Debugging

After I found out that the problem was with the synced folder feature in vagrant, I ssh-ed into the virtual machine and killed the command that was running on the horizon screen session of stack. The command was python manage.py syncdb --noinput. Once I killed the process, I got the following stacktrace.

The Stacktrace

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2013-10-04 04:25:09 Traceback (most recent call last):
2013-10-04 04:25:09   File "manage.py", line 11, in <module>
2013-10-04 04:25:09     execute_from_command_line(sys.argv)
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 453, in execute_from_command_line
2013-10-04 04:25:09     utility.execute()
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 392, in execute
2013-10-04 04:25:09     self.fetch_command(subcommand).run_from_argv(self.argv)
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 263, in fetch_command
2013-10-04 04:25:09     app_name = get_commands()[subcommand]
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 109, in get_commands
2013-10-04 04:25:09     apps = settings.INSTALLED_APPS
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 53, in __getattr__
2013-10-04 04:25:09     self._setup(name)
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in _setup
2013-10-04 04:25:09     self._wrapped = Settings(settings_module)
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 132, in __init__
2013-10-04 04:25:09     mod = importlib.import_module(self.SETTINGS_MODULE)
2013-10-04 04:25:09   File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
2013-10-04 04:25:09     __import__(name)
2013-10-04 04:25:09   File "/opt/stack/horizon/openstack_dashboard/settings.py", line 209, in <module>
2013-10-04 04:25:09     from local.local_settings import *  # noqa
2013-10-04 04:25:09   File "/opt/stack/horizon/openstack_dashboard/local/local_settings.py", line 92, in <module>
2013-10-04 04:25:09     SECRET_KEY = secret_key.generate_or_read_from_file(os.path.join(LOCAL_PATH, '.secret_key_store'))
2013-10-04 04:25:09   File "/opt/stack/horizon/horizon/utils/secret_key.py", line 55, in generate_or_read_from_file
2013-10-04 04:25:09     with lock:
2013-10-04 04:25:09   File "/usr/lib/python2.7/dist-packages/lockfile.py", line 223, in __enter__
2013-10-04 04:25:09     self.acquire()
2013-10-04 04:25:09   File "/usr/lib/python2.7/dist-packages/lockfile.py", line 248, in acquire
2013-10-04 04:25:09     os.link(self.unique_name, self.lock_file)
2013-10-04 04:25:09 KeyboardInterrupt

By looking at the most recent function that was called, it was waiting to create a lockfile. This lockfile creation process requires making a symlink and the call to make the symlink os.link appeared to be failed and no Timeout was passed into the function (that was bad, the timeout was None). There was a loop that caught the exception and retried infinetely. Then I tried to execute the same python command in an interactive python session and got a ‘Operation not permitted’ error. When I tried to manually link the file using ln command, it seemed to work just fine but for some reason, trying to do so using python doesn’t work.

The Python Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
def acquire(self, timeout=None):
    try:
        open(self.unique_name, "wb").close()
    except IOError:
        raise LockFailed("failed to create %s" % self.unique_name)

    end_time = time.time()
    if timeout is not None and timeout > 0:
        end_time += timeout

    while True:
        # Try and create a hard link to it.
        try:
            os.link(self.unique_name, self.lock_file)
        except OSError:
            # Link creation failed.  Maybe we've double-locked?
            nlinks = os.stat(self.unique_name).st_nlink
            if nlinks == 2:
                # The original link plus the one I created == 2.  We're
                # good to go.
                return
            else:
                # Otherwise the lock creation failed.
                if timeout is not None and time.time() > end_time:
                    os.unlink(self.unique_name)
                    if timeout > 0:
                        raise LockTimeout
                    else:
                        raise AlreadyLocked
                time.sleep(timeout is not None and timeout/10 or 0.1)
        else:
            # Link creation succeeded.  We're good to go.
            return

Since this is a permission issue, I tried to investigate the file ownership and permission information. They were owned by the vagrant user who tried to run the stack. But since this filesystem is a shared filesystem from the host machine using NFS the ownership and permission on the host machine also comes into account. Since the UID and GID of the vagrant user and my local username were different, the virtual machien couldn’t properly get access to the filesystem. That was the root cause!

Solution

The actual solution to fix the permission issue is to match the UID and GID of the vagrant user to the one on the local uesrname. I didn’t want to mess up my system by changing the UID of my machine. It might cause some unexpected troubles. I tried to change the UID and GID of the vagrant user on the virtual machine. It did work but I didn’t go with the solution since recreating the vagrant environment will again have the same issue and everytime I recreate the development environment, I’ll have to change the UID and GID of the vagrant user. Also if I change the host machine, the UID and GID will again change. So I decided not to use the synced folder approach. Instead, I had a copy of the /opt/stack directory on my host machine and wrote a small script using rsync which will update the code in the virtual machine using the one on my host machine. Once I make the code change, I’ll have to restart some services to see the change anyway. So running just another script didn’t seem to be a big deal for me. Here is the script I used.

Sample Sync Script

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash

echo "Syncing files.."
/opt/local/bin/rsync \
    -e ssh \
    -avrh \
    --progress \
    --update \
    --exclude 'stack-volumes-backing-file*' \
    /Users/kannanmanickam/Projects/openstack_dev/vm_code/stack \
    vagrant@192.168.27.100:/opt

echo "Sync complete!"

Note that in the script, I use the --update option. This option makes sure that the new files on the receiver are unaffected. Only the files locally changed will be updated on the receiver (virtual machine). For example, there will be new log files in the receiver which we don’t have in the host machine. These files will be unaffected. I also exclude the ‘stack-volumes-backing-file’. This file is used for backing the volumes created by cinder. This is usually 5 GB and there is no need for syncing his file. Before using this script make sure you have your public key available as an authorized key in the virtual machine so you don’t get prompted for password while running this script.

Overview

The yard generated documentation does not provide a way to specify the Google Analytics tracking ID in the configuration. If you want to have the tracking ID in the generated documentation, you have to go through all the files and inject the script. Also the files change every time the documentation is generated. I use the yard-generated documentation as the website for my jenkins_api_client project and want to use my Google Analytics tracking ID to see the usage of the project. So I decided to create a rake task that will do this for me

The Rake Task

This simple rake task will go through all html files in the subdirectories of current directory and inject the google analytics tracking script. Include this rake task in your Rakefile and replace UA-XXXXXXXX-X with the google analytics tracking ID of your website.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
task :apply_google_analytics do
  # The string to replace in the html document. This is chosen to be the end
  # body </body> tag. So the script can be injected as the last thing in the
  # document body.
  string_to_replace = "</body>"
  # This is the string to replace with. It include the google analytics script
  # as well as the end </body> tag.
  string_to_replace_with = <<-EOF
    <script type="text/javascript">
      var _gaq = _gaq || [];
      _gaq.push(['_setAccount', 'UA-XXXXXXXX-X']);
      _gaq.push(['_trackPageview']);

      (function() {
        var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
        ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
        var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
      })();
    </script>
  </body>
  EOF

  files = Dir.glob("**/*.html")

  files.each do |html_file|
    puts "Processing file: #{html_file}"
    contents = ""
    # Read the file contents
    file =  File.open(html_file)
    file.each { |line| contents << line }
    file.close

    # If the file already has google analytics tracking info, skip it.
    if contents.include?(string_to_replace_with)
      puts "Skipped..."
      next
    end

    # Apply google analytics tracking info to the html file
    contents.gsub!(string_to_replace, string_to_replace_with)

    # Write the contents with the google analytics info to the file
    file =  File.open(html_file, "w")
    file.write(contents)
    file.close
  end
end

The Rake task in action

1
bundle exec rake apply_google_analytics

The rake task will go through all

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Processing file: doc/_index.html
Processing file: doc/class_list.html
Processing file: doc/file.README.html
Processing file: doc/file_list.html
Processing file: doc/frames.html
Processing file: doc/index.html
Processing file: doc/JenkinsApi/CLI/Base.html
Processing file: doc/JenkinsApi/CLI/Helper.html
Processing file: doc/JenkinsApi/CLI/Job.html
Processing file: doc/JenkinsApi/CLI/Node.html
Processing file: doc/JenkinsApi/CLI/System.html
Processing file: doc/JenkinsApi/CLI.html
Processing file: doc/JenkinsApi/Client/BuildQueue.html
Processing file: doc/JenkinsApi/Client/Job.html
Processing file: doc/JenkinsApi/Client/Node.html
Processing file: doc/JenkinsApi/Client/System.html
Processing file: doc/JenkinsApi/Client/User.html
Processing file: doc/JenkinsApi/Client/View.html
Processing file: doc/JenkinsApi/Client.html
Processing file: doc/JenkinsApi/Exceptions/ApiException.html
Processing file: doc/JenkinsApi/Exceptions/CLIError.html
Processing file: doc/JenkinsApi/Exceptions/CrumbNotFound.html
Processing file: doc/JenkinsApi/Exceptions/Forbidden.html
Processing file: doc/JenkinsApi/Exceptions/ForbiddenWithCrumb.html
Processing file: doc/JenkinsApi/Exceptions/InternalServerError.html
Processing file: doc/JenkinsApi/Exceptions/JobAlreadyExists.html
Processing file: doc/JenkinsApi/Exceptions/JobNotFound.html
Processing file: doc/JenkinsApi/Exceptions/NodeAlreadyExists.html
Processing file: doc/JenkinsApi/Exceptions/NodeNotFound.html
Processing file: doc/JenkinsApi/Exceptions/NotFound.html
Processing file: doc/JenkinsApi/Exceptions/NothingSubmitted.html
Processing file: doc/JenkinsApi/Exceptions/ServiceUnavailable.html
Processing file: doc/JenkinsApi/Exceptions/Unauthorized.html
Processing file: doc/JenkinsApi/Exceptions/ViewAlreadyExists.html
Processing file: doc/JenkinsApi/Exceptions/ViewNotFound.html
Processing file: doc/JenkinsApi/Exceptions.html
Processing file: doc/JenkinsApi.html
Processing file: doc/method_list.html
Processing file: doc/top-level-namespace.html

Since this is newly generated documentation, the file has to go through all files inject the google analytics script in all of them. If the file already has the analytics script, that file will be skipped. A generated page using this rake task can be found in the jenkins_api_client project documentation here.

The .irbrc file

The .irbrc file is usually placed in your home directory (~/.irbc). This file is just a regular ruby file and you can write any arbitrary ruby code that you want to be executed before the IRB session is loaded.

The .pryrc file

The .pryrc file is very similar to the .irbrc file and is placed in your home directory (~/.pryrc) and this file is loaded before the Pry session is started.

Customizing the session

If you use a particular gem every time you are in an IRB or a Pry session, you can simply include that in your .irbrc or .pryrc file so you do not have to manually require that gem every time you enter the session. An example for this would be the pretty print (pp) library. Just add the require statement in the rc file and that will load this library whenever you enter the session.

1
require "pp"

Note: The following customizations are IRB specific and Pry has most of them in-built.

If you use IRB instead of Pry, you will have to install some additional gems to get the coloring enabled in the output. Pry has this in-built so you do not have to do any additional work. To enable coloring in IRB, the wirble gem should be installed. The following simple configuration will enable coloring using the wirble gem.

1
2
3
4
5
6
7
require "wirble"

# Initialize Wirble
Wirble.init

# Enable colored output
Wirble.colorize

Tab completion in IRB can be enabled by

1
require "irb/completion"

Automatic indentation can be set by

1
IRB.conf[:AUTO_INDENT] = true

If you would like to save the history between IRB sessions, it can be achieved by

1
2
3
4
5
6
require "irb/ext/save-history"
# Number of lines/commands to save

IRB.conf[:SAVE_HISTORY] = 100
# The location to save the history file
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.irb-save-history"

Per-project customization

The .[irb|pry]rc file can be customized to automatically load the environment for your project based on your current working directory. Whenever I work with Ruby projects, I keep a separate window open for my pry session. I do not want to load and initialize the project every time I enter te session. I usually have the credentials saved separately in a YAML file. So in the .pryrc file, I get the current working directory and based on the directory, I load and initialize my project. The object I initialize in the rc file will be available inside the irb/pry session, which I can then use for my interactions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Obtain the present woking directory
current_dir = Dir.pwd
# Customizations required for my jenkins_api_client project. The block of
# code inside this if block will load the gem project and then initialize
# the client.
if current_dir =~ /^(.*?\/jenkin_api_client)/
  # YAML is required to parse the credentials from the login.yml file
  require"yaml"
  # Obtain the library path of your ruby project
  path = "#{$1}/lib"
  # Insert this path to the front the $LOAD_PATH
  $LOAD_PATH.unshift(path)

  # Load the project as a rubygem
  require "jenkins_api_client"
  # Initialize the client by loading credentials from the yaml file
  @client = JenkinsApi::Client.new(
      YAML.load_file(File.expand_path("~/.jenkins_api_client/login.yml"))
  )
  puts "logged-in to the Jenkins API, use the '@client' variable to use it"
# Custmoizations required for my jenkins_launcher project
elsif current_dir =~ /^(.*?\/jenkins_launcher)/
    ...
elsif current_dir =~ /^(.*?\/some_other_project)/
    ...
end

Now that I have my rc file setup to prepare my project when I am in my jenkins_api_client project directory, when I enter the Pry session, my project is loaded and ready for me to play with. If I change my directory to a different project and enter irb/pry session again, that project will be loaded instead.

1
2
3
4
Kannans-MacBook-Pro:jenkins_api_client kannanmanickam$ pry
logged-in to the Jenkins API, use the '@client' variable to use the client
Welcome back to the Pry session, Kannan!
[1] pry(main)>

It is as simple as adding another condition to match your project and name do the initializations to load your project. The above method will work even if you are deep inside your project directory structure as the regex we use to match th current directory uses non-greedy approach.

What is Pry?

Pry is a feature-rich alternative to the standard IRB (Interactive Ruby Shell) which provides tons of features including syntax highlighting, source code browsing, documentation browsing, dynamic code reloading, live editing of code and much more. More information about Pry can be found in their website. In this post I am going to describe how I use Pry for building and debugging my projects with examples as screenshots.

Features

Methods cd and ls

Once you are in the Pry session, you can change your scope to any Object you want. During the development of my project jenkins_api_client I often use interactive Pry session to debug the code. Please take a quick look at my project for easier understanding of the examples here. My .pryrc initializes my project by setting up the credentials.

The @client instance variable is an object to the Client class. For example, if I want to list all jobs available in jenkins, I have to access that using @client.job.list_all. The @client object has a method called job which will initialize the Job class and return an object which can then be used for calling methods on the Job class. With the use of Pry, you can easily change the scope so you do not have to use the chain of objects.

The cd command (just like changing the directory in Unix systems) can be used to change the scope into the object specified as the argument. The ls command can be used to list (just like listing files in Unix systems) all resources available in current scope. If you want just the methods just do ls -m. Now that we have changed the scope to the job object, we can call the methods on that object by simply calling the method by the method name.

Methods show-method and show-doc

The show-method can be used for displaying the method. Similarly, the show-doc method can be used for displaying the documentation of the given method.

The show-method can also be used for system methods. For example, the rm_rf method in the FileUtils class can be viewed simply by calling show-method FileUtils#rm_rf.

Editing of methods and live reload of code

If you are working on a method and want to edit and see the behavior for the change, you can simly use the edit method and specify the method name. It will open your default editor. Since I have my default editor set to vim, this example opens up vim in the shell itself.

In the following screenshot, you can see that the editor is opened and the cursor is pointing to the correct method.

Once the the edit is completed and the file is saved, the editor is closed and you will be returned to the Pry session. As Pry automatically reloads code for the class the method resides, if you execute the method again, the code changes will be reflected.

Note: Do not wonder why there is a red line at the 80 character mark. I follow the ruby style guide of limiting character width to 80 characters. I use the colorcolumn feature in Vim.

Bingo! The code change can be seen clearly.

These examples are just a few that are used for development and debugging. It provides much more functionality. Check out more examples and screencasts on pryrepl.org