Delivering perf reports

A teammate and I are developing a generic scheduling system to replace the built-in Kubernetes CronJob scheduler for triggering thousands of periodic jobs. The built-in Kubernetes scheduler is too slow for our needs – it takes about ten minutes for it to iterate through the list of schedules, but some of our jobs need to run more frequently than once every ten minutes.

This post isn’t about schedulers, though. It’s about considering a performance characteristic report to be a deliverable. When future developers eventually reach the point of needing to scale, it will be valuable to know which scaling knobs can be turned and how. And presenting the report early in the project lifecycle helps spread understanding to outside stakeholders.

By performance report I mean a quantitative exploration of how the proposed system would behave in response to variable constraints and loads. For example, we had to first demonstrate that our new scheduler would be able to handle the current workload (otherwise, there’d be no reason to consider it). Then we simulated changing factors outside of our control, like increasing the number of jobs and the duration of those jobs. We also simulated changing aspects of the system itself, like the number of worker threads. Aside from describing the service’s response to stress, this exercise directly influenced how we architected the system with an eye toward scalability.

I wouldn’t recommend doing this kind of stress test for all projects all the time, though. It might be suitable when the load on the system is anticipated to grow substantially in the medium term. Or if the expected load is going to be stable, but something about the proposed system makes the performance profile hard to estimate, like if it’s implemented on a different stack from what the team is familiar with. Conducting scaling experiments could be overkill for systems that are based on familiar technologies and that are expected to have stable/unremarkable load. In our case, the project’s existence was spurred by degrading performance, so it made sense to keep scaling at top of mind.

We’re shipping this performance characteristic report right alongside the software. Results are captured and explained in a Jupyter notebook that we demoed at one of our company’s internal weekly demo days. We think that the new scheduling system can support 20x our current load – twice the 10x goal we had for the project. If and when future engineers anticipate the need to scale, this report will be there for them. It will show them approximately when the current implementation’s limits will be reached, and may provide insight into what stop-gap measures can be used before an entirely new solution is needed.

Getting URL and Tab Title from Firefox with AppleScript

A long time ago, a coworker shared an AppleScript function that gets the title and URL from the current tab in Chrome and formats the result as an Emacs org mode link.

Org mode links have the format [[url][title]], e.g.:

[[https://matthewbilyeu.com/blog/][· Matt's programming blog]]

Invoked via Alfred, this script was helpful for getting links from Github pull requests, Trello cards, etc. into my org TODO list. When I began using Firefox Quantum, however, the Chrome-specific script was of no use.

Firefox does not have AppleScript support, which means that any automated interaction with the application must be clumsily achieved through simulating GUI events. I found out that such GUI-based automation is prone to failure. It took some trial and error and cobbling various online snippets together, but so far this script has been fairly robust:

use scripting additions
use framework "Foundation"

tell application "Firefox" to activate

-- get the tab title from FF
tell application "System Events" to tell process "firefox"
	set frontmost to true
	set the_title to name of windows's item 1
	set the_title to (do shell script "echo " & quoted form of the_title & " | tr '[' ' '")
	set the_title to (do shell script "echo " & quoted form of the_title & " | tr ']' ' '")
end tell

set thePasteboard to current application's NSPasteboard's generalPasteboard()
set theCount to thePasteboard's changeCount()

-- send cmd+l and cmd+c keystrokes to FF to highlight and copy the URL
tell application "System Events"
	keystroke "l" using {command down}
	delay 0.2
	keystroke "c" using {command down}
end tell

-- wait for the clipboard content change to have been detected
repeat 20 times
	if thePasteboard's changeCount() is not theCount then exit repeat
	delay 0.1
end repeat

-- get the clipboard contents
set the_url to the clipboard

return "[[" & the_url & "][" & the_title & "]]" as text

References:

• https://apple.stackexchange.com/questions/271161/how-to-get-the-selected-text-into-an-applescript-without-copying-the-text-to-th
• https://www.snip2code.com/Snippet/69440/AppleScript-to-get-url-from-Safari--Fire
• https://www.alfredforum.com/topic/2013-how-to-get-frontmost-tab%E2%80%99s-url-and-title-of-various-browsers/

Tech debt

For a long time I’d considered technical debt to be just a bad thing. That it was a mess left behind by imprudent coders of the past. I hadn’t paused to reflect on the idea of it.

Only recently did I come to appreciate that tech debt can be a tool, just like monetary debt. Monetary debt is a (hopefully) temporary suboptimal state that’s intentionally entered with an eye toward a long-term goal. E.g., you take on debt to pay for a university degree that you think will result in you having a higher salary in the future. The debt is useful in this way. Tech debt, too, can be useful.

This became clear to me while working on a prototype to test market fit for a potential new feature at work. By making tradeoffs that sacrificed design robustness for speed of implementation, we intentionally introduced tech debt into the system. Designing and implementing a polished, scalable system before knowing whether the offering was valuable to our customers could have been an expensive mistake. Tech debt is allowing us to explore an idea and quickly react to evolving requirements.

Of course, just as with monetary debt, too much tech debt can be disastrous. Tech debt is also harder than monetary debt to measure. Still, it’s a useful tool in some scenarios, and I’m glad I have a more nuanced understanding of it.

Command Lining

I’ve slowly accumulated knowledge of a few tricks that help me to be productive at the terminal prompt. These are things that I somehow missed during years of computer science schooling and my early career. I often learn things, then immediately forget the time before I learned them, and then take that new information for granted. I’m trying to reflect on such learning and distill the best parts. And so hopefully someone who happens upon this list may find a new tip or two.

TLDR

TLDR - Simplified and community-driven man pages.

This isn’t a built-in bash command, but I’ve found it to be very handy. It’s one of the first programs I install on a new system. Type tldr <command> to see usage examples of the most common options of command line programs.

Tree

Another not-built-in program. Imagine if ls recursively displayed a hierarchical directory listing. I like visual information, so I like tree.

tree -L 3
.
└── uploads
    ├── 2013
    │   ├── 11
    │   └── 12
    ├── 2014
    │   ├── 03
    │   ├── 06
    │   ├── 07
    ├── 2015
    │   └── 01
    ├── 2016
    │   ├── 01
    │   └── 04
    ├── 2017
    │   └── 07
    └── 2018
        ├── 01
        └── 02

Return to previous directory

I happened to be reading an intro to bash commands once, and I was surprised that one of the first lessons showed a use of cd that I somehow wasn’t familiar with. This is how you can change the working directory back to the previous working directory. Great for hopping between a couple related projects.

cd -

Curly brace expansion

This is a really cool feature of bash and one I wished I’d learned about earlier. Read more about it here.

for i in {0..4}; do echo $i; done
0
1
2
3
4

My favorite use of brace expansion is to expand two long path names. E.g.

cp ~/my/long/path/to/a/file.txt ~/my/long/path/to/a/file.txt.bak

can be rewritten as

cp ~/my/long/path/to/a/file.txt{,.bak}

Previous command

!! will give you access to the previous command. Of course you can also press the up arrow, but this is useful for modifying the previous command. 99% of the time I use this it’s because I forgot sudo.

touch somefile.txt
touch: somefile.txt: Permission denied

sudo !!

sudo touch somefile.txt

Search command history

Press ctrl + r to start searching backwards through the command history. This is another one I wish I’d learned sooner.

(reverse-i-search)`touc': touch somefile.txt

And the entire command history is available via history:

history | tail -n3
13637  ls
13638  ll
13639  history | tail -n3

User Feedback

I spent some of my free time over the past three weeks rebuilding a small single page app I maintain: Tumblr Top (code here).

The original incarnation of this app was written circa 2015 in CoffeeScript and AngularJS 1.4. The new version is in React and Semantic UI. There are also a few simple charts to visualize blog post and tag popularity that were coded using the Victory charting library.

I rewrote the app as an excuse to use React on a personal project and to make the site easier to build and maintain going forward. But the original version was running smoothly with something like 2-3k users per month. I didn’t want to upset the regular users by fixing what wasn’t broken.

So I built out the updated site to what I thought was a reduced but perhaps satisfactory level of functionality. I hosted the new app at a different url, and added a callout to the old site asking users to try it out.

While the new app was in-progress, I invited the early adopters to leave comments and criticism. Google Forms made soliciting and collecting user feedback dead simple (alternatives: SurveyMonkey or Typeform). The feedback survey was to the point. It only had four questions:

• Do you like the redesign?
• Do you miss the [missing feature]?
• Were there any errors or bugs with the redesign? (If so, please provide your browser / operating system)
• Any other feedback about the redesign or the app in general?

Over the course of a couple weeks the survey received 19 responses. This was enough to help me massively improve the first cut. The answers to those four questions gave me a lot of insight:

• The performance improvements I’d added had come at the cost of my API access often being rate-limited
• There was a critical display bug in a browser I hadn’t tested
• Other small bugs were highlighted
• Users had become very accustomed to what I thought were unimportant design details of the original version
• A feature I thought was useless was meaningful to a large percentage of users
• Strangers on the internet are willing to fill out a survey
• You can’t please everybody

I’m thankful to the handful of power users who took time out of their lives to provide valuable feedback for the app. Without that survey I may have shipped a shoddy update and degraded my users’ experience.