Pythonspeed.com Website Review

When your CI is taking forever on AWS EC2, it might be EBS

You’re running you test suite or your Docker image packaging on a EC2 server. And it’s slow. docker pull takes 15 minutes just to verify the images it ed in 1 minute. apt or dnf installs take another 15 minutes. pip install or conda install take even more time. It’s a fast machine with plenty of memory—in fact, you may have be using a custom machine precisely because Travis CI or Circle CI builders are underpowered. And yet it’s still taking forever. Why? Quite possibly the problem is with your EBS disk’s IOPS.

When your CI is taking forever on AWS EC2, it might be EBS

IOPS as bottleneck

Tests too slow? You can make them faster—

Stuck with slow tests? Speed up your feedback loop

You submit a pull request, the CI system kicks off the tests—and then you wait. And wait some more. And by the time it turns red to tell you there’s a problem you’ve moved on to something else. What were you doing again? And eventually, hours or even days later, you finally get your code merged. For a single developer all this waiting and context switching is expensive; for a whole team it’s that much worse. Obviously what you need is a faster test suite. But what if you can’t make your test suite faster? You don’t have the time, or don’t have the skills, or maybe you’ve optimized it so much you don’t think you can do any better. Are you just stuck wasting your time? Don’t give up: there is something you can do. What you can do is speed up the feedback loop of your test system. In this article I’ll explain: Why faster feedback is what really matters. An overview of some of the ways you can speed up your testing feedback loop. Why you need a faster feedback loop When your CI systems runs your tests, there are two possible outcomes: Success. Failure. If your tests p successfully you can probably just merge your code—in fact some software teams will have the branch merge automatically if it’s p ed review and tests. There’s not much to do, really, and if it takes a while to get there that’s not ideal, but it’s also not that bad. If your tests fail after 30 minutes, however, you will need to: Stop whatever new task you switched to. Try and remember what that failing code did. Figure out the problem and fix it. Switch back to your new task. Remember what you were doing on your new task. And this may repeat multiple times if your fix was insufficient or problematic and the tests fail again (as they often do). In short, a failing CI run is a much bigger timesink than a successful run, because of all the expensive mental context switching it causes. In particular, failing tests are expensive when it takes 10 minutes, 30 minutes, or even 2 hours until you know that your code needs to be fixed. If you can speed up the time-to-reported-failure of your test suite, if you can get meaningful feedback faster, you can reduce or eliminate the mental context switching. If you push code and almost immediately get told your tests are failing, you can just fix them immediately, and repeat until you’re fairly certain that everything is going to p . Some ways to speed up your feedback loop Even if you can’t speed up your test suite overall, then, you can still make failures occur as quickly as possible. It’s much more useful if your CI run fails within 3 minutes than if it fails after 90 minutes. Here are some of the ways you can do that: Linters and code yzers The first thing your test suite should do is run a linter. The linter can catch obvious problems before the relevant tests do, so if it runs first, fails, and then ends your CI run, you’ll be notified of the problems faster. Personally I’m a fan of pylint (when it’s configured correctly), but you can also use flake8 or other tools. And mypy or other type checking tools can help catch problems if you’re using type annotations. You can run these tools quite early—depending on how you’ve configured them, before you’ve installed any dependencies, allowing your CI run to provide feedback even faster. Run relevant tests first If you changed module water.py, chances are the tests that will fail are in test_water.py and test_bucket.py, not in test_steel.py. By running relevant tests first, you can increases the chances of failing quickly. There are a variety of ways to do this, from manually recording the dependency information (which is how the Twisted project does it) to heuristic tools like py.test-testmon. You should still run the whole test suite, but only after running the relevant tests. If you can’t avoid it, running the same tests twice (once because they are relevant, and then a second time when you run the full test suite) may still be worth it for the faster feedback loop. Run faster and smaller tests first If you have two sets of tests, fast small scale tests (one of the usages of “unit test”) and slow integration tests, run the fast tests first. With any luck they’ll catch the problem before you get to the slow integration tests. Speed up the rest of your development process Tests are just part of your development process, and you might be able to speed up the feedback loop elsewhere as well. For example, if you currently do code reviews after tests p , maybe instead you want to do them immediately when the pull request is submitted, in parallel with CI. Faster feedback, faster development The suggestions above are just the beginning; likely there are others way you can optimize for feedback in your particular situation. Just remember: it’s the speed of feedback that matters, and the easiest way to speed up feedback is to have your test suite find relevant failures as quickly as possible. The faster your feedback loop, the less need there is for context switching—and the faster you’ll be able to ship features and bug fixes.

Stuck with slow tests? Speed up your feedback loop

Why you need a faster feedback loop

Linters and code yzers

Why Pylint is both useful and unusable, and how you can actually use it

This is a story about a tool that caught a production-impacting bug the day before we released the code. This is also the story of a tool no one uses, and for good reason. By the time you’re done reading you’ll see why this tool is useful, why it’s unusable, and how you can actually use it with your Python project.

Why Pylint is both useful and unusable, and how you can actually use it

Pylint saves the day

Tests too slow? You can make them faster—

Articles

Faster code can give you a higher profit margin, more users, and faster development cycles. Learn how to speed up your Python application and test suite.

Articles

Speed up your test suite

Privacy Policy

Faster code can give you a higher profit margin, more users, and faster development cycles. Learn how to speed up your Python application and test suite.

Privacy Policy

Personal Information We Collect