Hunting down phantom mock instances in XCTestCases

Sharing a simple lesson and insight learnt when chasing a mysterious issue

Image is from Unsplash

No stub for method `XXX()`

In my workplace, recently we faced occasional test errors that happened only when run in test suite, even though individual test runs fine.

No stub for method `XXX()` using parameters (). (NSInternalInconsistencyException)

Certainly we have this Cuckoo stub somewhere in our test, but not in the failing test.

This kept me scratching my head for some time, but recently I found out that it is basically happening due to leak of the mock objects (and other objects stems from tests). If you are as naive iOS developer as me, you would expect that the mock object generated in your test will be purged automatically after each test (we are in post-ARC era, why not??), but that is not actually the case.

I have created a small sample project to reproduce this error. You can take a look at the project if you want, but you can just read through this article and get the gist as well.

The structure of the project is very simple. A ViewModel instance has a reference to Logger instance. ViewModel listens to UIApplication.didBecomeActive event and calls Loogger's logAppDidBecomeActive() method.

Class diagram of sample project

Now, let’s look at the test file ViewModelTests.swift:

We have two tests

• testAInit()
• testAppDidBecomeActive()

The first test just initializes ViewModel instance and make sure nothing strange happens. The second test broadcastsUIApplication.didBecomeActiveNotification and make sure that the ViewModel calls Logger's logAppIsActive() method. It seems simple and everything looks fine, right?
However, if you execute above test, it will fail at line 32 with error

No stub for method `logAppIsActive()` using parameters (). (NSInternalInconsistencyException)

Isn’t it strange though, considering that we are certainly mocking LoggerProtocol.logAppIsActive() just above?? (line 28)

Why?

It turns out that it is not the mock from the second test, but the mock from first test is receiving logAppIsActive() method as well. So, the issue will go away if we mock logAppIsActive() method in the first test as well. But why does this happen?

After it failed at the second test, the memory debugger looks like this :

The blue part is from the host app, which you can ignore. One notable thing is that you have two ViewModelTests instances (yellow) and also two instances of ViewModel (red), one from the first test, one from the second test respectively. Apparently, the VieModel from the first test is leaking here. Because of that, when second test issues UIApplication.didBecomeActive notification, the first ViewModel receives it as well and calls its MockLoggerProtocol instance which doesn’t stub its logAppIsActive method => fail.

Why why??

The key is the fact that we have two ViewModelTests instances in this case. My misunderstanding was that XCTest would generate XXXTests instances only whenever they run, and would discard it after finishing. As a matter of fact, it seems XCTest will create all the test instances at the beginning of running test suite and keep them in memory. (For example, if you have 100 test classes who has 10 test cases for each, XCTest will generate 100x10=1000 XXXTests instances at launch and keep them in memory.) This is the memory debugger right after starting the test suite, before even executing the first test:

You can see that the two ViewModelTests instances are already there (one for the first test, one for the second test). XCTest keeps these test instances while the suite is running, so all the instance variables in the test instances will live through and cause memory leak.

Solution

The solution for this is very simple. We just need to nil out all the instance variables of the test instances. Like

override func tearDownWithError() throws {
    logger = nil
    sut = nil
}

I have seen that whenever Apple demonstrates test classes in their doc or presentation, they nil out instance variables, but didn’t really think about it, almost thought it is just a convention. Apparently that is a very meaningful/important step. Even though I focused on an issue related to Cuckoo’s mock object, the principle applies to plain `XCTest` cases too. If you don’t nil out the instance variables of your test classes, your objects will leak and 1. may cause unexpected behavior when running the test suite, 2. eventually may bring memory related issues as the tests scale (slow test run ~ crash).

Conclusion

The short lesson from this article is that you have to always nil out instance variables of your test instance in tearDown. Especially when you face No stub for method XXX() message that doesn’t make sense, the first culprit you should look at is this.

A slightly extended lesson is to have a mental model that all test instances are generated at first and kept in memory throughout the test suite run. I don’t know why Apple designed it like this (doesn’t scale well, I guess), but that’s how it is right now.

Hope you enjoyed the read, happy coding!!