Weeknotes 65

Taking inspiration from Rowan and Alice, I’ve separated my weeknotes (and solitary yearnotes) from my other, less frequent posts (e.g. “Using a Raspberry Pi for Time Machine”) on the homepage.

As well as the existing combined Atom feed, there are now separate feeds for posts and weeknotes if you only care to subscribe to one.

In preparation for a launch and an expected burst of traffic this week, I worked on automatically scaling a Rails application hosted on Heroku.

Heroku provide their own autoscaling but only for their “performance” tier of dynos which are 10 times the price of their standard ones.

Thankfully, they recommend third-party add-ons such as Rails Autoscale if you’re not using performance dynos or if your app has variable response times (as most do). I hadn’t heard of Rails Autoscale before but the demo was very compelling. Its use of Heroku “request queueing” time rather than response time makes it especially useful when your app might have the occasional spike in response time without impacting its ability to serve requests to other clients.

Adam McCrea’s “How Many Heroku Dynos Do You Need, and Which Size—An Opinionated Guide” is a fantastic guide and led me to look into “in-dyno concurrency” with multiple Puma workers rather than relying solely on threads.

However, as soon as I bumped the number of workers up for my application, a slow memory leak became much more pronounced with my app peaking at 131.5% of its memory quota.

Fortunately, I stumbled across Frederick Cheung’s “Debugging a Memory Leak in a Rails App” which not only described how to find memory leaks in running Rails applications using rbtrace and Sam Saffron’s excellent guide but described a potential memory leak in Rails’ prepend_view_path.

What did I find in a before_action in my Application Controller? Something like the following:

def prepend_subdomain_specific_templates
  prepend_view_path(Rails.root.join('app', 'views', request.subdomain))
end

As Frederick points out, prepending a String or Pathname to the view path causes Rails to initialize a new Action View Resolver for every request. This would be fine as Ruby should garbage collect the Resolvers when they are no longer used but Rails’ template caching means that each Resolver defines methods whose names are never collected, causing a slow but steady leak. If you’re calling prepend_view_path on a particularly heavily trafficked route, this leak will become more pronounced.

The fix in my situation was to initialize a Resolver per subdomain once and then re-use the same Resolver whenever possible, e.g.

SUBDOMAIN_SPECIFIC_TEMPLATES = {
  'subdomain1' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain1'),
  'subdomain2' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain2'),
  'subdomain3' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain3')
}.freeze

def prepend_subdomain_specific_templates
  prepend_view_path(SUBDOMAIN_SPECIFIC_TEMPLATES.fetch(request.subdomain))
end

With the fix deployed, the resulting memory usage graph made me very happy indeed.

I’ve been rightly chastised for using the trope of interpreting children’s TV shows through the eyes of a jaded adult before but I’m genuinely enjoying “In the Night Garden…”.

Watching C’s excitement build as we watch Igglepiggle lie down in his boat, covering himself with its sail as we pan up from the rolling ocean to see the stars of the night sky bloom into flowers is a delight.

The creators’ intentions behind the series is interesting too:

“We became very aware of the anxiety surrounding the care of young children which manifested itself in all kind of directions – but the one big subject that came up again and again was bedtime. It’s the classic time for tension between children who want to stay up and parents who want them to go to bed.

“We wanted to explore the difference between being asleep and being awake from a child’s point of view: the difference between closing your eyes and pretending to be asleep and closing your eyes and sleeping.

“So this is a programme about calming things down whereas most children’s TV is about gee-ing everything up!”