12 Testing, debugging, and fixing code

Prerequisites

Before reading this chapter, you are recommended to have read Chapter 2

We’ve written a lot of Julia code so far, and while some errors have occurred, but with a heavy dose of foresight, we’ve been able to explain them away quite quickly. If you’re following along and have tried out some of the code along the way, or written some of your own, you’ll inevitably have run into more. In reality, exceptions happen all the time, and tracing down their causes is a large part of the skill of programmming. In this chapter, we’ll explore some of the ways that can aid us.

12.1 The stacktrace

When an exception is raised, the first step is to work out what caused it. Sometimes the simple error message is enough, but not always. Let’s use the following recursive function as an example:

sum1ton(n) = n == 1 ? 1 : sum1ton(n-1) + n

sum1ton (generic function with 1 method)

For some inputs, this seems to work fine:

sum1ton(20000)

200010000

However, for others, we get an error:

sum1ton(200000)

ERROR: StackOverflowError:

If you’re familiar with recursive functions, you’ll know what the error is immediately, but for the sake of demonstration let’s keep digging. Along with every error message, we get a stacktrace, which thus far we’ve omitted from error messages in this book since they can be quite long, but for this section, we’ll include them. The stacktrace lists out the functions that we were in the middle of executing when the exception happened. In the case of sum1ton, our stacktrace might look like this:

Stacktrace:
 [1] sum1ton(n::Int64) (repeats 79984 times)
   @ Main .\REPL[1]:1

This tells us that the error happened in the function sum1ton, since it is at the top of the stacktrace. It also tells us what the method is that it was using (from the type signature), and where that method is defined (from the second line, REPL[1] means the first line of the REPL). The most pertinent information in this instance, however, is the phrase (repeats 79984 times). A StackOverflowError happens when there are too many nested function calls, which in this case is 79984, so we can’t use sum1ton with an input as large as 200000

We can therefore tell from the stacktrace where the exception object was created (by whichever function is listed at the top), and the sequence of functions called running up to the error. However, it doesn’t necessarily tell us where the problem is, and we might have to look to the error message, or work it out ourselves. For example:

sqrt(-1)

ERROR: DomainError with -1.0:
sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).
Stacktrace:
 [1] throw_complex_domainerror(f::Symbol, x::Float64)
   @ Base.Math .\math.jl:33
 [2] sqrt
   @ .\math.jl:677 [inlined]
 [3] sqrt(x::Int64)
   @ Base.Math .\math.jl:1491
 [4] top-level scope
   @ REPL[1]:1

The top listed function in the stacktrace here is Base.Math.throw_complex_domainerror, which as the name suggests, is simply a function to create the Exception object and throw it. There’s nothing broken about that, so we can look down the stacktrace to see where the problem might lie. In this case, the error is caused by the input of -1 into sqrt, which as the error message explains, should be Complex if we expect a Complex result.

12.2 Adding error detection to code

While the stacktrace of an exception can tell us where an error happened, it isn’t always all that helpful in telling us why. For that, we may need to try something else.

12.2.1 Simple functions and macros

One very quick and easy way to diagnose an error is to add extra lines of code which output values at various points throughout the code. These could be intermediate calculated values, variables that you want to check the scope of, or simply printed words telling you that the program has reached that point.

Our old favourite function print is excellent for this purpose, as wherever we add it in, we’ll get an output to stdout (usually the REPL) with whatever value we put into it. Often, even better is the function display. For example, if we are using the Plots package, and we want to debug a Plot, then print would be quite useless to us, as it won’t show us what the Plot looks like as a plot. display, in contrast, will:

using Plots
p = plot()
print(p)

Plot{Plots.GRBackend() n=0}

display(p)

The display function is intended to show its input in the most descriptive way possible. For a Plot, this is naturally to show it as an image, so that’s what it will do. Meanwhile, the purpose of print is to turn anything into a String format and print that text into the REPL, which is not always the most helpful.

Another option is the macro @assert. If we follow @assert with code that returns a Bool, then if we get true, nothing will happen, but if we get false, an AssertionError will be thrown:

@assert 1 == 1.0

@assert 1 === 1.0

ERROR: AssertionError: 1 === 1.0

We can also follow our code by a String which will serve as an error message:

@assert 1 === 1.0 "One's an Int64, one's a Float64"

ERROR: AssertionError: One's an Int64, one's a Float64

This can naturally be interspersed into your code to check if the expected result of your calculations matches with the actual result. Similarly, we can do type assertions, which work much like type declarations in method definitions that we met in Chapter 6, and the syntax is correspondingly similar:

1::Int64

1::Float64

ERROR: TypeError: in typeassert, expected Float64, got a value of type Int64

1::Real

You can also add your own exceptions into code which will work just like the default ones, for diagnostic purposes within our own programs. The simplest way of doing this is the error function, which takes a String as argument that will become an error message:

error("an error has occurred.")

ERROR: an error has occurred.

There are also various Exception types, allowing special errors to be constructed with more information, much of which are used in the common errors that we see. Exception itself is the abstract type that is the parent of all of them, so we can list some out as its subtypes:

# filter gives only the elements of the collection where the function returns true
filter(T -> isdefined(Base, Symbol(T)), subtypes(Exception))

31-element Vector{Any}:
 ArgumentError
 AssertionError
 BoundsError
 CanonicalIndexError
 CapturedException
 CompositeException
 ConcurrencyViolationError
 DimensionMismatch
 DivideError
 DomainError
 EOFError
 ErrorException
 InexactError
 ⋮
 OverflowError
 ProcessFailedException
 ReadOnlyMemoryError
 SegmentationFault
 StackOverflowError
 StringIndexError
 SystemError
 TaskFailedException
 TypeError
 UndefKeywordError
 UndefRefError
 UndefVarError

You should choose the one that most closely describes your error. The function throw then serves to cause the Exception to occur, stopping the program and giving the message.

e = BoundsError("message", 10)

BoundsError("message", 10)

throw(e)

ERROR: BoundsError: attempt to access 7-codeunit String at index [10]

Throwing an Exception will need to be combined with some sort of control flow, usually conditionals, so we can detect if an error has occurred, and raise an exception if needs be.

x = -1
x > 0 || throw(DomainError(x, "x must be positive"))

ERROR: DomainError with -1:
x must be positive

12.2.2 Comments, documentation, and example outputs

The larger a project is, the harder it is to keep track of the code within it. The best way to prevent code from being an undecipherable mess is by commenting, which we’ve seen before is done with #

# "This is a comment, so nothing will happen"

"This isn't a comment, so it will be returned"

"This isn't a comment, so it will be returned"

Comments can be notes about what a particular line or section of code does, but they can also be ‘commented-out’ code, that is lines of code that you can have run or not run at will by simply adding or removing a #:

x = 1
# x += 1
x

x = 1
x += 1
x

Many IDEs will have a keyboard shortcut for toggling commented code (for instance, on VSCode, the default is Ctrl-/), since it is such a common thing to want to do.

For objects such as functions, macros, and types, we can do more than just a comment to describe the function of the whole thing. Instead, we can use a docstring, which is very simply a String written on the line (or lines, for a multiline String) immediately before the definition, telling us what functionality the structure has and how we might use it.

"""
    Element(name::Symbol, sym::Symbol, z::Int64)

Define an element of the periodic table.
"""
struct Element
    name::Symbol
    sym::Symbol
    z::Int64
end

Element

Convention

There are a multitude of conventions that you are supposed to follow for docstrings. However, there’s really no need unless you’re creating code that will be used by a lot of people; if it’s just for you then the best docstring is the one that you understand best.

The type Element works just the same as normal here, since although the String is evaluated with Julia code, Julia doesn’t have anything to do with its value, so it gets ignored:

Element(:uranium, :U, 92)

Element(:uranium, :U, 92)

However, there is a way to return a docstring, which we’ve met a long time ago: the REPL’s help mode! Indeed, when given a function, macro, or type name, all that help mode does is search for the definition of whatever you’re looking up, and return the corresponding docstring (or docstrings, as there can be several, e.g. for different methods of the same function). Equally, the @doc macro provides the same service outside of the REPL:

@doc Element

Element(name::Symbol, sym::Symbol, z::Int64)

Define an element of the periodic table.

Another thing that you may wish to include in comments in your code are example outputs. If you know one of a group of functions isn’t working, but you can’t work out from the stacktrace alone which is broken, then you can test some example outputs that you have commented. We’ll see more of this idea with the Test module later in the chapter.

Note

It’s simple enough to include examples in the docstring, and indeed you can find many functions and the like which do in Base. In fact, many use the Documenter package to combine these examples with testing capabilities like those which we’ll look at with Test shortly.

12.2.3 The `try` block

As we’ve mentioned several times, exceptions aren’t necessarily a bad thing, as they can help to highlight where problems have occurred. The issue with them is that they always stop our program running, so we can’t see what influence this error will have on the rest of the program, or even test the rest of the program to find other problems.

The solution to this, and many other similar problems, is the try block. Like the if statement, it comes with many pieces, which we need to understand individually:

try
    [...] # A
catch e
    [...] # B
else
    [...] # C
finally
    [...] # D
end

A try block always begins with try. Any code in this piece (marked # A) will be run as normal, except that if an error occurs, it won’t be raised immediately
The next piece is catch (marked # B), which catches any error thrown within the try piece. Above, we have followed catch by a variable name e, which will be assigned the value of the Exception object that was thrown. Note that if you don’t need that object, the variable name after catch can be omitted. The block of code within the catch piece will then run, but only if an error occurred in try
If no error occurred in try, then the code in the else piece (marked # C) will run instead
Whether an error occurred or not, the finally piece will run last. This is usually intended for necessary actions, such as closing files after we’re done with them (as we did in Chapter 10)

The reason we’ve chosen to describe these four section of the try block as pieces is that we have quite a bit of choice in fitting them together. While the order must stay the same as the above, we can remove some of the pieces if we don’t need them. There are three rules we need to follow:

We have to start with try (it is a try block, after all)
There must be at least one of catch and finally following try
else can only be used if catch is also used

Let’s now put try to the test. As we saw earlier, the sqrt function doesn’t allow square roots of negative numbers unless those numbers have first been converted to a Complex type:

sqrt(-1)

ERROR: DomainError with -1.0:
sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).

There’s a good reason for this (it’s type stability), but let’s circumvent it. We’ll write our own function with a try loop so that we can detect this DomainError before it stops the execution:

function bettersqrt(n)
    try
        return sqrt(n)
    catch e
        # We want to preserve any other Exception
        e isa DomainError || throw(e)
        return sqrt(complex(n))
    end
end

bettersqrt (generic function with 1 method)

We can try the bettersqrt function out, to make sure it works on positive and negative inputs as expected

bettersqrt(1)

1.0

bettersqrt(-1)

0.0 + 1.0im

If we deliberately cause a different error, then this should pass through unchanged:

bettersqrt("1")

ERROR: MethodError: no method matching sqrt(::String)
Stacktrace: [...]

caused by: MethodError: no method matching sqrt(::String)
Stacktrace: [...]

The extra caused by: bit of the error message here tells us the original error that happened under try. In this case, it’s the same error since we threw it back straight away, but had we thrown a different error from within catch, that would have taken the place of the first MethodError.

Warning

While using the try block is tempting to stop exceptions from occurring, remember that it’s usually better to fix the problems rather than just putting them in a try block and forgetting about them!

Note

As with any new block we introduce, it’s useful to know its scope implications. Each piece of the try block functions like a for loop with regards to scope, independently of each other, so any variable assigned a value within them is local to that piece alone, unless it was already defined before the try block, and the catch variable that we’ve called e consistently is always local to the catch piece.

12.3 The `Test` module

The final method we’ll look at for testing is the Test module, which provides a couple of macros that can help with automatically testing our code.

using Test

To demonstrate its capabilities, we’ll investigate two simple calculation problems. The first problem we’ll look at is to determine whether a given year is a leap year or not. We’ll solve this with a function called isleapyear, which by the magic of Julia, is expressed in simply one line of short-circuited conditionals:

function isleapyear(y::Int64)
    y % 400 == 0 || y % 100 != 0 && y % 4 == 0
end

isleapyear (generic function with 1 method)

Now let’s write some tests that our function needs to pass. We can do this with the @test macro from Test, which we’ll give an expression that has a Bool output that we want to be true. For instance, here are five examples that we need isleapyear to pass:

@test isleapyear(2000)

Test Passed

@test !isleapyear(2001)

Test Passed

@test !isleapyear(2023)

Test Passed

@test isleapyear(2024)

Test Passed

@test !isleapyear(2100)

Test Passed

Fantastic, it passed all the tests! What happens if it fails one?

@test isleapyear(2023)

Test failed at [...]
  Expression: isleapyear(2023)

ERROR: There was an error during testing

Note

Obviously this test failed because it is wrong, not the function, and this is always important to keep in mind!

If we want to make tests that we can more easily run again and again, we can have the @test macro give the return value of a function:

leapyeartest() = @test isleapyear(2024)

leapyeartest (generic function with 1 method)

leapyeartest()

Test Passed

To combine multiple tests together, we could put several in the same function, but we would only see the result of the last one since only the last value would be returned. Instead, we can use the @testset macro, which will run multiple @test tests and summarise the results.

Let’s use @testset on a slightly trickier problem, calculating the day of the week from a year, month, and day. This time, we’ll write the tests first, in the form of a function so we can repeatedly call it. @testset has several options for formatting the various @tests; we’ve demonstrated using a begin-end block as well as a for loop here:

function daytests()

    finals = [
        (1930, 7, 30),
        (1954, 7, 4),
        (1970, 6, 21),
        (2010, 7, 11),
        (2022, 12, 18)
    ]

    @testset "Date tests" begin
        @testset "Scientist's birthdays" begin
            @test day(1743, 8, 26) == :Monday
            @test day(1776, 4, 1)  == :Monday
            @test day(1799, 5, 21) == :Tuesday
            @test day(1822, 7, 20) == :Saturday
            @test day(1867, 11, 7) == :Thursday
            @test day(1912, 6, 23) == :Sunday
        end
        @testset "World Cup final $(final[1])" for final ∈ finals
            # The World Cup final is always on a Sunday, right?
            @test day(final...) == :Sunday
        end
    end
end

daytests (generic function with 1 method)

Note that to do this, we’re prescribing several things about the solution. The function must be called day, must take three Int64 inputs in the order year, month, day, and must output a Symbol for the corresponding day of the week. We could have chosen differently for each of these, so it’s important to make sure that a prewritten testing function like this aligns properly with your function that you want to test.

Now, we need write the function day. This is a fun problem to try and solve for yourself, but here’s one solution:

const DAYS = [:Monday, :Tuesday, :Wednesday, :Thursday, :Friday, :Saturday, :Sunday]
const DAYS_BEFORE_MONTH = [0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334]

function day(y::Int64, m::Int64, d::Int64)

    # Offset from 1st January 2000
    Δy = y - 2000
    nleaps = Δy < 0 ? -count(isleapyear.(y:1999)) : count(isleapyear.(2000:y-1))
    Δd = 365Δy + nleaps + DAYS_BEFORE_MONTH[m] + d + (isleapyear(y) && m > 2) - 1
    
    # 1st January 2000 was a Saturday, so Δd = 0 should give index 6
    DAYS[mod(Δd + 6, 1:7)]

end

day (generic function with 1 method)

We can now run the tests, using the testing function that we have set up:

daytests()

World Cup final 1930: Test Failed at [...]
  Expression: day(final...) == :Sunday
   Evaluated: Wednesday == Sunday

Stacktrace: [...]
Test Summary:           | Pass  Fail  Total  Time
Date tests              |   10     1     11  0.2s
  Scientist's birthdays |    6            6  0.0s
  World Cup final 1930  |          1      1  0.2s
  World Cup final 1954  |    1            1  0.0s
  World Cup final 1970  |    1            1  0.0s
  World Cup final 2010  |    1            1  0.0s
  World Cup final 2022  |    1            1  0.0s
ERROR: Some tests did not pass: 10 passed, 1 failed, 0 errored, 0 broken.

The output table tells us that there was only one problem out of our 11 tests, and it happened with the World Cup final 2022 input. The problem here is not with our function, that actually works flawlessly. The issue is that the 1930 World Cup final was, bizarrely, on a Wednesday, so our test was actually wrong!

It can also be useful to look at the timings in the rightmost column. For our simple functions here, this is unenlightening, but for more complicated functions, it may be helpful to trace down inefficiencies in functions that do work, but are very slow to run.

Note

While some would bring up the efficiency of Julia code as one of its key selling points (and indeed, it is often very efficient), we’ve barely mentioned speed of computation throughout this book. The reason for this is simple, speed is always secondary to having your code do what you want it to, no matter how long it takes.

For long calculations, or large datasets, writing efficient code is very desirable, but as for how that is done, we’ll leave for others to explain, e.g.:

https://docs.julialang.org/en/v1/manual/performance-tips/: The official Julia page on improving performance
https://viralinstruction.com/posts/optimise/: A blog post by Jakob Nybo Nissen on optimising Julia code (written for Julia 1.7.3, but remains relevant to 1.9.3)

12.1 The stacktrace

12.2 Adding error detection to code

12.2.1 Simple functions and macros

12.2.2 Comments, documentation, and example outputs

12.2.3 The try block

12.3 The Test module

12.2.3 The `try` block

12.3 The `Test` module