use two integer variables called i and sum
assign the value 0 to both variables

while i is less than 100 do:
  increase i by one
  add value of i to sum

optionally print the final value of sum

• Nim is a multi-paradigm programming language. Unlike some popular programming languages, Nim doesn’t focus on the OOP paradigm. It’s mainly a imperative and procedural programming language, with varying support for OOP, data-orientated, functional, declarative, concur- rent, and other programming styles. Nim supports common OOP features, including inheritance, polymorphism, and dynamic dispatch.

• The generated executables are dependence free and small: a simple chess program with a plain GTK-based graphical user interface is only 100 kB in size and the size of the Nim compiler executable itself is about 6.5 MB. It is possible to shrink the executable size of "Hello World" programs to about 10 kB for use on tiny microcontrollers.

• Nim is fast. Generally performance is very close to other high-performance languages such as C or C++. There are some exceptions still: other languages may have libraries or applications that have been tuned for performance for many years, while similar Nim applications are so far less tuned for performance, or maybe are more written with a priority of short and clean code or run-time safety.

• Clean Python-like syntax with significant white-space, no need for block delimiters like {} or begin/end keywords, and no need for statement delimiters like ;

• Safety: Nim programs are type- and memory-safe — memory corruption is prevented by the compiler as long as unsafe low level constructs like casts, pointers and the addr operator or the {.union.} pragma are not used.

• Fast compiler. The Nim compiler can compile itself and other medium-size packages in less than 10 seconds, and upcoming incremental compilation will increase that speed further.

• Nim is statically typed: each object and each variable has a well-defined type, which catches most programming errors already at compile time, prevents run-time errors, and ensures highest performance. At the same time the statically typing makes it easier to understand and to maintain larger code bases.

• Nim supports various memory management strategies, including manually allocations for critical low-level tasks as well as various garbage collectors including a destructor based, fully deterministic memory manager.

• Nim produces native, highly optimized executables and can also generate JavaScript output for web applications.

• Nim has a clean module concept which helps to structure large projects.

• Nim has a well-designed library which supports many basic programming tasks. The full source code of the library is included and can be viewed easily from within the HTML-based API documentation.

• Library modules like the OS module provides OS independent abstractions, which allows to compile and run the same program on different operating system without modifications.

• The Nim standard library is supported by more than 1000 external packages for a broad range of use cases.

• Asynchronous operation, threading and parallel processing is supported.

• Nim supports all popular operating systems like Linux, Windows, MacOS and Android.

• Usage of external libraries written in C is easy and and occurs directly without any glue code, and Nim can even work together with code written in other languages, for example there are some Nim <-> Python interfaces available.

• Many popular editors have support for Nim syntax highlighting and other IDE functionality like on-the-fly checking for errors and displaying detailed information about imported functions and data types.

• In the last few years Nim has reached some important milestones: Version 1.0 with some stability promises was released, and with the ARC and ORC memory management strategies and full destructor support fully deterministic memory management comparable to memory management in C++ or Rust is available. So problems of conventional garbage collectors like delayed memory deallocation or longer pausing of programs due to the GC process are gone. And some larger companies have started using Nim in production, the most important may be currently the Status Corp. with their Etherium client development.

We mentioned already that Nim is a multi-paradigm programming language, that supports various programming styles. While we may regard Nim in first line as an imperative, procedural programming language, it supports the popular functional and object-orientated programming styles well.

In classical OOP programming languages, we have the concept of classes with attributes, and methods that are very closely bound to the classes, as in Python:

In this Python snippet, we declare a class User, with a custom method named say() bound to this class. Then we create an instance variable of this class and call its say() method.

This tight bounding of methods to classes is not very flexible, e.g. extending the set of methods of a class may be difficult or impossible. Another problem with such a class concept is, that it is not always clear to which class a method belongs when more than just one single class is involved: Imagine that we need a method that appends a single character to a text string. Is that method a member of the character class, or a member of the text string class?

Nim avoids such a strict class concept, while its generalized method call syntax allows us to use a class like syntax for all of our data types: e.g. to get the length of a string variable, we can write len(myString) in classic procedural notation, or we can use the method syntax myString.len() or just myString.len. The compiler regards all these notations as equivalent, so we have the method syntax available without the restrictions of the class concept. The method call syntax can be used in Nim for all data types, even for plain numbers — so the notations abs(myNum) is fully equivalent with myNum.abs.

The Python code from above may look in Nim like

Instead of the classes we use object types in Nim, and we define procedures and methods that can work on objects or other data types.

As an example for the functional programming style in Nim we may look at some code fragment from a real world app that has to generate a string from four numbers, separated by commas. Using the mapIt() procedure imported from the sequtils module and the fmt() macro from the strformat module, we may write that in functional programming style in this way:

In the imperative, procedural style we would write it like

Nim is a compiled and statically-typed language. While for interpreted, dynamically-typed languages like Python we have to run every statement to check even for trivial errors, the Nim compiler checks for most errors during the compile process. The static typing together with the well-designed Nim type system allows the compiler to catch most errors already in the compile phase, like the undefined addition of a number and a letter, and to report the errors in the terminal window or directly in the editor or IDE. When no errors are found or all errors have been fixed, then the compiler generates highly optimized dependency free executables. And this compilation process is generally really fast, for example the compiler compiles itself in maybe 10 to 30 seconds on a typical modern PC.[8]

Modern concepts like zero-overhead iterators, compile-time evaluation of user-defined functions and cross-module inlining in combination with the preference of value-based, stack-located data types leads to extremely efficient code. Multi-threading, asynchronous input/output operations (async IO), parallel processing and SIMD instructions including GPU execution are supported. Various memory management strategies exists: selectable and tuneable high performance Garbage Collectors (GC), including a new fully deterministic destructor based GC, are supported for automatic memory management. These can be disabled for manual memory management. This makes Nim a good choice for application development and close-to-the-hardware system programming at the same time. The unrestricted hardware access, small executables and optional GC will make Nim a perfect solution for embedded systems, hardware driver and operating system development.

Nim offers a modern type system with templates, generics and type inference. Built-in advanced data types like dynamic containers, sets, and strings with full UTF support are completed by a large collection of library types like hash tables and regular expressions. While the traditional Object-Oriented-Programming programming style with inheritance and dynamic dispatch is supported, Nim does not enforce this programming paradigm and offers modern concepts, like procedural and functional programming. The optional method call syntax allows to use all data types and functions in an OOP like fashion, e.g. instead of len(myStr) we can also use the OOP style myStr.len.[9] The powerful AST-based hygienic macro system offers nearly unlimited possibilities for the advanced programmer. This macro and meta-programming system allows compiler-guided code generation at compile time, so the Nim core language can be kept small and compact, while many advanced features are enabled by user defined macros. For example the support of asynchronous IO operations has been created with these forms of meta-programming, as well as many Domain Specific Language (DSL) extensions.

The Nim compiler and all modules of the standard library are implemented in Nim. All source code is available under the less restricted MIT license.

The Nim forum is hosted at:

https://forum.nim-lang.org/

and the software running the forum is coded in Nim.

Real-time chat is supported by IRC, Gitter, Discord, Telegram and others.

Nim is also supported by Reddit.com and Stackoverflow.com:

Started more than 12 years ago as a small community project of some bright CS students led by Mr. A. Rumpf, it is now considered as one of the most interesting and promising programming languages, supported by countless individuals and leading companies of the computer industry, for instance, it’s actively used in the areas of application, game, web and crypto-currency development. Nim has made a large amount of progress in the last few years: it reached version Nim v1.6 with some stability guarantees and a new deterministic memory management system was introduced, which will improve support of parallel processing and the use of Nim in the area of embedded systems development.

Nim was created by Mr. A. Rumpf in 2008, supported by a few volunteers. Finally, in 2018 Nim got some significant monetary support by Status Corp. and in 2019 the stable Nim version 1.0 was released. But still Nim is developed by a small core team and some volunteers, while some other languages like Java, C#, Go, or Rust are supported by large companies, or like C and C++ have a very long history and well-trained users. And finally there are many competing languages, some with a longer history, and some maybe better suited for special purposes, like JavaScript, Dart or Kotlin for web development, Julia or R for numeric applications, or Zig, C and Assembly for the tiny 8-bit microcontrollers with a small amount of RAM.

Nim is already supported by more than 1000 external packages which cover many application areas, but that number is still small compared to really popular languages like Python, Java or JavaScript. And some Nim packages can currently not really compare with the libraries of other languages, which have been optimized for years by hundreds or thousands of full-time developers.

Indeed the future of Nim is not really secure. Core developers may vanish, financial support may stop, or maybe a better language may appear. But even if the development of Nim should stop some day, you will still be able to use it, and many concepts that you may have learned with Nim can be used with other modern languages too.

When you use C as your first language, you may learn well how computers really work, but the learning experience is not that nice, progress is slow and C lacks many concepts of modern programming languages. C++, Rust or Haskell are really too difficult for beginners. So currently many starts with Python. While you can learn high level concepts well with Python and you get useful results fast, you learn not much about the internal working of computers. So you may never understand why your code is slow and consumes so much resources, and you will have no idea how to improve the program or how you could run it successfully on restricted hardware. It’s like learning to drive a car, without any knowledge about how a combustion engine, the transmission, or the brakes really work. Nim has none of these restrictions, as we have high level concepts available like in Python, but we have access to low level stuff too, so we can really understand the internal workings, if we want. Learning resources for Nim are still not that good as for mainstream languages, but there exists some nice tutorials already, and hopefully this book will help beginners also a bit.

Generally yes, in the same way as Pascal was in the 1980’s, and Modula/Oberon were at the end of the last century. But Nim still has the same problems as the wirthian languages: They do not really help with finding a job. When we teach the kids some JavaScript or C, they may find at least a simple employment when they have to leave the intended education path early for some reason. With niche languages this is unfortunately not the case, so teachers should know about their responsibility. And of course teaching against the interests of the kids makes not much sense. When they want to learn some JavaScript to make some visual effects or whatever easily, then it is hard to teach another language which may be not immediately available on the PC at home or their smartphone.

Maybe not. When you intend to learn a programming language today and make a great video game tomorrow, then definitely not. This is just not possible. While there are nice libs for making games with Nim already available, there exists easier solutions in other languages. With some luck you may find some source code for that languages, so that you can patch a few strings and maybe modify some colors and the background music and call it your game.

Nim is a quite universal language, so it is a good candidate for someone who intends to learn only one single language. But of course it is always a good idea to learn a few other languages later. Generally we can not really avoid learning C, as so much C code exists world wide. Most algorithm that have ever been invented are available as a C implementation somewhere, and most libraries are written in C or have at least a C API, which you can use from other languages including Nim. As C is a small language without difficult language constructs, some minimal C knowledge is generally sufficient to convert a C program to another language. Often that conversion process is supported by tools, like the Nim c2nim tool. So learning some C later is really a good idea, and when you have some basic understanding of Nim and CS in general, learning some C is an easy task. Learning C before Nim would be an option still, as for C more learning resources exists. So years ago some people recommended learning C or Python before Nim. But Nim has enough learning resources now, so we recommend indeed starting with Nim directly.

Maybe it is just not the ideal solution for you. A racing bicycle or a mountain bike are both great devices, but for cycling a few hundred meters to the bakers shop both may be not the perfect solution. A plain old bicycle would do better. Even as Nim seems to join the benefits of a racing bicycle and a mountain bike well — high performance and robust design — and is not expensive, it is just not the optimal solution for everybody. People who write only tiny scripts and have not to care about performance can continue using Python. People who are only interested in special applications, maybe only in web development or only in tiny 8 bit microcontrollers may not really need Nim. Nim can do this and much more well, but for special use cases better suited languages may still exist. And someone who has managed to learn C++ really well over a period of many years may decide to continue with C++ also. Currently another possible reason for not using Nim can be missing libraries. When you need some important libraries for your project, and these are currently not available for Nim, this can be of course a big problem in the case that you have not the skills or the time to write them from scratch or at least create high level bindings to a C library.

var sum: int
var i: int
sum = 0
i = 0
while i < 100:
  inc(i, 1)
  inc(sum, i)
echo sum

cd
mkdir mynimfiles
cd mynimfiles
gedit test.nim

ls -lt
cat test.nim

nim c test.nim

./test

nim c test.nim

nim c -d:release test.nim

nim c -d:danger test.nim

nim c -d:release --gc:arc -d:useMalloc --passC:-flto --passC:-march=native board.nim

const Pi = 3.1415

var velocity: int

const SquareOfFive = 5 * 5
echo(5 * 5, SquareOfFive) # ordinary procedure call
echo 5 * 5, SquareOfFive # command invocation syntax

var a: int
a = 2 + 3
echo a
echo(cos(0) + 2)

const
  Pi = 3.1415
  Year = 2020
var
  sum: int
  age: int

var a: int
echo a; inc(a) (1)
a = 2 * a + (2)
  a * a

var
  sum, age: int

var
  year: int = 1900

var
  year = 1900

var a, b: unsigned int
a = 3
b = 7
a = a - b

var
  a: int = 0
  b: int

const
  Max = 100
var
  sum, i: int
while i < Max:
  inc(i)
  inc(sum, i)
echo sum

echo "Please enter some text"
var mytext = readLine(stdin)
echo "you entered: ", mytext

from terminal import getch

stdout.write("May you tell me your name: ")
var answer = readLine(stdin)
if answer != "no":
  echo "Nice to meet you, ", answer
echo "Press any key to continue"
let c = getch()
echo "OK, let us continue, you pressed key:", c

We have already used the int data type, which indicates a signed integer type of 4 or 8 byte size, depending on the operating system. The reason why the size of that type depends on the word size of the OS will become clear later, when we explain what references and pointers are.

Beside the int data type, Nim has some more data types for signed and unsigned integers: int8, int16, int32 and int64 are signed types with well-defined bit and byte size, and uint8, uint16, uint32 and uint64 are the unsigned equivalents. The number at the end of the type name is the bit size; we get the byte size when we divide that value by 8. Additional we have the type uint, which corresponds to int and has same size, but stores unsigned numbers only. [15] Generally we should try to use the int type for all integral numbers, but sometimes it can make sense to use the other types. For example, when you have to work with a large collection of numbers, you know that each number is not very big, and your RAM is not really that large, then you may decide for example to use int16 for all your numbers. Or when you know that your numbers will be really big and will not fit in a 4 byte integer, then you may use the int64 type to ensure that the numbers fit in that type even when your program is compiled and executed on a computer with a 32 bit OS.

For integer numbers we have the predefined operators +, - and * available for addition, subtraction and multiplication. Basically these operations works as we may expect, but we have to remember that we may get overflows. For signed ints we get compile- or run-time errors in that case, while unsigned ints just wrap around, see example at the end of this section. For division of integers we have the operators div, mod, and / available. The div operator does an integer division ignoring the remainder, mod is short for modulus and gives us the remainder of the division, and / finally is currently only predefined for the signed int type and gives us a fractional result of data type float. That type is introduced in the next section.

Remembering how div and mod behaves when the divisor or dividend are negative can be confusing, and it may differ for other programming languages. You may find a detailed justified explanation for the concrete behaviour in the Nim manual and at Wikipedia.

When performance matters we generally should try to use the "CPU native" number type what for Nim is the int type. And we should try to avoid using math expressions with different types, as the CPU may have to do type conversion in that case before the math operation can be applied. Adding two int8 types can on some CPU’s be slower than adding two ints, because the CPU may have to size extend the operands before the math operation is performed. But this depends on the actual CPU, and there are important exceptions: Multiplying two ints would result in an int128 result if int size is 64 bit, which can be slow when the CPU does not support that operation well. Another important point to consider for maximum performance is the cache usage. If you are performing operations on a large set of data, then you may get a significant performance gain when large fractions of your data fits in the caches of your computer, as cache access is much faster than ordinary RAM access. So using smaller data types, i.e. int32 instead Nim’s default int which is int64 on a 64 bit OS may increase performance in this special application.

When we use Nim on tiny microcontrollers, maybe even on 8-bit controllers like the popular AVR devices, it may be best to use only integers of well defined size like int8.

When we write integer literal numbers, then we use generally our common decimal notation, as in var i = 100. To increase the readability for long number literals we can use the underscore character as in 1_000, that underscore character is just ignored by the compiler. We can also write integer literals in binary, octal or hexadecimal notation. For that we prefix the literal value with 0b, 0o or 0x. The leading zero is necessary, and the next letter indicates binary, octal or hexadecimal encoding. But such integer literal notation is very rarely used.

More important is the actual size of integer literals, in particular when we use type inference. Ordinary integer literals have the int type, but integer literals not fitting in 32 bit have int64 type. We can also specify the type of integer literals by appending the literal with i8, i16, i32 or i64 for signed types and with u, u8, u16, u32 or u64 for unsigned types. We can separate the actual number and the suffix with a ' character, but that is not necessary for the integer literals.

In arithmetic expressions integer types of different sizes are generally compatible when all the types are signed or unsigned, e.g. in the example code from above we could write echo a + b + c, and typeof(a + b + c) is int64, that is the expression is propagated to the largest type of all the involved operands. But echo a + b + c + d would not compile, as for such a mix of signed and unsigned operands it is not clear if signed or unsigned arithmetic should be used. Also note that even on a 64 bit OS echo typeof(a) is typeof(b) would print false.

An important property of the current Nim implementation of A. Rumpf used with the C backend is the fact that unsigned integers does not generate overflow errors but simple wrap around:

Above code would print the numbers 0 up to 127 and then terminate program execution due to an overflow error. But when we change the data type to uint8 we would get a continues sequence of the numbers 0 up to 255. After the value 255 is reached, the value wraps around to 0 again and the process continues. This behavior can lead to strange bugs and is the reason that the Nim team generally recommends to avoid unsigned integers.

For compatibility with external libraries Nim has also the integer types cint and cuint, which exactly match the C types int and uint when we compile for the C or C++ backend. For the JavaScript backend, the LLVM backend or other backends these types may be also available, for details you should consult the compiler documentation. For most operating system and C compilers the int and uint types in C are 4 bytes in size, but there can be exceptions, so we better should not write code that depends on the actual byte size of the types. The Nim types cint and cuint are generally only used for parameter lists of © library functions. To match other integer types like C char, short, long, longlong Nim supports these types when we put a c letter in front of the name like clong. Again you should consult the Nim compiler manual when you need more details, i.e. when you create bindings to external libraries.

Another important numeric data type is float, for floating point numbers. Floats are an approximation of real numbers. They can also store fractions, and are most often printed in the decimal system with a decimal point, or in scientific notation with an exponent. Examples for the use of variables of float data type are

The variable mean is assigned the result of the division of two float literals — the result has again the data type float, and so the compiler can infer for the type of variable mean that same type. If we printed the result of the division there would be a decimal point and some digits following it. For variable x we specify the float type explicitly and assign the value 12. We could use type inference if we assigned 12.0, because the compiler can recognize by the decimal point that we want a float, not an int variable. In line 3 we use scientific notation for the float literal that we assign to y, and the assigned value is 1.2 * 10^3 = 1200.0. Literal values like 2E3 are also valid float literals — the value would be 2000.0. But literals with a decimal point and no digits before or after the point — 1. or .2 — are not valid in Nim.

In the current Nim implementation float variables always occupy 64 bits. Nim has also the data type float64 which is currently identical to plain float, and float32 which can store only smaller numbers and has less precision.[16] Floats can store values up to a magnitude of approximately 1E308 with a positive or negative sign, and floats have a typical precision of 16 digits. That is, when you do a division of two arbitrary floats and print the result, you will get up to 16 valid digits. If you would try to print more than 16 significant digits, then the additional decimal places would be just some form of random garbage. Note: The number of significant digits of a floating point number is the total number of digits before and after the decimal point, but possible leading zero digits would not be counted. The reason that leading zeros are not significant is just that in the ordinary notation of numbers we always assume that there is just nothing before the first non zero digit. For our car odometer 001234.5 km is identical to 1234.5 km. And if we give our body size as 1.80 m or 180 cm makes no difference, both values have 3 significant digits.

Generally we use floats whenever integers are not sufficient for some reason. For example when we have to do complicated mathematical operations which include fractional operands like Pi, or when we have to do divisions and need the exact fractional value.

The float, float32 and float64 data types provides the +, -, * and / operators for addition, subtraction, multiplication and division. Unlike for the int types, for the float types we get never overflow or underflow errors, and also no error for a division by zero. But the result of an operation of two float operands can be a special value like system.Inf, system.NegInf or system.NaN. The first two indicate an over- or underflow, and NaN (Not a Number) indicates that the result of an operation is not a valid number at all, for example the result of a division by zero or the result of calculating the square root of a negative number. This behaviour is sometimes called saturated arithmetic. When a variable has one of these special values, and we apply further math operations, then this value is kept. So we can detect at the end of a longer mathematical calculation if something went wrong — we have not to check after each single operation.[17] An interesting property of floating point numbers is, that when we test two variables of float type for equality, and one has the value NaN, then the test is always false. That is the test a == NaN is always false. When we forget this fact, we may initialize a float variable to the value NaN and later test with if a == NaN: to check if we have already assigned a value, which is not what we really had in mind, as that test has always a negative result. The actual test for the value NaN is a == a, which is only false when a has the value NaN, or we may use math.isNaN(). More useful constants and functions for the float data types can be found in the std/fenv module, and functions working with floats like the trigonometric ones are available from the std/math module.

For floats we have the operators +, -, * and / for addition, subtraction, multiplication and division. For powers with integral exponents you can use the ^ operator, but you have to import it from the std/math module. The expression x ^ 3 is the same as x * x * x. The math module contains many more functions like sin() or cos(), sqrt() and pow(). The function name sqrt() is short for square-root, and pow() stands for power, so pow(x, y) is x to the power of y, when both operands have type float. For performance critical code you should always keep in mind that pow() is an actual function call, maybe a call of an dynamic library which can not be inlined, so a call of pow(x, 2) may be a lot slower than a plain x * x. And even when using the ^ operator as in x ^ 3 we should be a bit critical. But of course we always hope that the compiler will optimize all that for us.

The operators +, -, * and / can be used also when one operand is a float variable and the other operand is an int literal. In that case the compiler knows that we really intend to do a float operation and converts the int literal automatically to float type. But when one operand is a float variable and the other is an int variable, then an explicit type conversion is necessary like in float(myIntVal) * myFloatVal. One explanation why in this case the int value is not automatically converted to float is, that this may mean a loss in precision, as large int64 values can not be presented as an float. Well, for int32 this reason does not really apply, but still there is no automatic conversion. But indeed as Nim is used as a systems programming language, it seems to be a good decision to need explicit conversions in this case, as it makes it more clear what really is intended. And generally we should try to avoid to use a lot operations with mixed types, as that may make type conversions necessary, which may cost performance. If we really do not care, we may import the module std/lenientOps, which defines the arithmetic operations for mixed operands.

Float literals have the float data type by default, but as for integer literals we can also explicitly specify the data type: The suffixes f and f32 specify a 32 bit float type, and d and f64 specify a 64 bit type. We can separate the suffix from the actual number with a ' character, but that is not required as long as there is no ambiguity. We can also specify float literals in binary, octal or hexadecimal notation, when we append one of these suffixes. In case of hexadecimal notation, the ' is obviously needed to separate the suffix, as f and d are valid hex digits.

As for integer variables Nim supports also the compatible types cfloat and cdouble which match the C types float and double when the C backend is enabled. For most C compilers C float matches Nim’s float32 and C double matches Nim’s float64.

Two important properties of floats are that not all numbers can be represented exactly and that math operations are not absolutely accurate. Recall that in our decimal system, some fractions like 1/2 can be represented exactly as 0.5 in decimal notation, while others like 1/3 can be only approximated as 0.3333…​ As all data, floats are stored internally in binary form following the IEEE Standard for Floating-Point Arithmetic (IEEE 754). In that format some values, e.g. the value 0.1, can not be represented exactly. As a result, some simple arithmetic operations, executed in the computer, will give us not exactly that result that we may expect. As we should really remember this important fact, we will investigate this behaviour with a small example program where we divide a few small integer numbers after conversion to float by another to float converted integer n and sum the result n times:[18]

which generates this output:

The echo() procedure prints up to 16 significant digits for a float value, and so the accumulated tiny arithmetic errors become visible. After our remarks above that should be not surprising any more, the general solution is to round results to less than 16 decimal digits before printing. Various ways to do that will be shown later in the book. A related issue of float arithmetic is caused by scaling and extinction. When we add numbers with very different magnitudes, the result may be just the value of the largest number, as in echo 1.0 == 1.0 + 1e-16 which prints true. The tiny summand is just too small to actually chance the result, that is as when you switch on a torch on a sunny day, it will not really become brighter. Maybe more surprising is, that calling echo() with some simple float literals will print a different value, like echo 66.04 which gives 66.04000000000001 for Nim v1.6, while with Python3 we get 66.04 exactly. But indeed that is only surprising for people who do not understand well what a statement like echo 66.04 really does: We know already, that the value 66.04 is converted by the compiler to an internally binary representation, and then converted back to a decimal string when we run the program. So it is not that surprising that in this process some tiny inaccuracies can accumulate. Actually it is possible to get exact 16 digits precision when a very smart conversion routine like the ryu or dragonbox algorithm is used.

From the discussions above it should be clear that testing two floats for equality is often problematic. Instead for just testing for equality we may better define a small epsilon value like eps = 1e-14 and then write (a - b).abs < eps. Seems to be not bad, and can be seen often and works often, but not always. Imagine you write a program which processes chemical elements and you work with atomic mass and radii. So maybe the result with above test is that all the atoms of the periodic table have equal mass and equal size, at least when you should use the SI system with meter and kilogram as base units. So a equality test like

may be a better solution in the general case. Whenever you need a general equality test you should think about the problem and do some tests — the code above are just untested possible examples.

At the end of this sections some remarks about the performance of float data types compared to plain ints: On modern hardware like the popular x86 systems for the basic operations performance of floats and ints is very similar, addition, subtraction and even multiplication is basically done in only one clock cycle, and division may be a bit slower. Even operations like sqrt() which have been regarded as slow in the past are now close to a plain addition on modern hardware. As the CPU does its float arithmetic internally with 64 or even with 80 bit, float32 is not faster than float 64, as long as the operations are not memory bound, that is large data sets are processed, so that it is an advantage when the data types are smaller so that more of it fits into the cache. For tiny microcontrollers and embedded devices things are very different, as these devices generally have no floating point units. So the compiler has to emulate all the float arithmetic, maybe by use of libraries. This is very slow and produces large executables. So when writing software for modern desktop PCs, there is no reason to try to avoid float math, when solving the problem with float is easier. When the data extends a very width range, e.g. from nm to millions of km, or when operations like square root or trigonometric functions are needed, then there is generally no way and reason to avoid float. In the case that float or ints may work both, it is generally a good strategy to try to use ints as first try. Ints may still provide better performance for SIMD, threading and parallel processing, as ints may avoid the expensive saving of floating point CPU registers. For restricted hardware we should better try to avoid float math. But all this is a difficult topic, and these advice can give you only some simple recommendations, which may be wrong for a concrete case. So finally you have to decide yourself, and as always it is a good idea to do some performance tests.

References:

Before we continue with subrange types we should introduce the distinct types. In the real world we have a lot of quantities for which the set of meaningful math operations is restricted and which should not be mixed with quantities of other types. For example we may have the quantities time and distance measured in seconds and meters and mapped to the float or int data type. While adding seconds and adding meters is a valid operation, adding seconds to meters makes no sense and would be a program bug if it should occur in the program code. But again dividing a distance by a time period resulting in the average speed would be a valid operation. Nim provides the distinct keyword which allows us to define new data types that are based on existing types, but that are not compatible with them or with other distinct types. And the new defined distinct types have no predefined operations, we have to define all desired operations our self.

For distinct types we have to define all the allowed operations our self. We can convert distinct types to the base types and then use operations of the base type or we can borrow operations from the base type by use of the {.borrow.} pragma. Using distinct types can be complicated when the new type should support many operations, but it can make our code more save. For some data type with a very limited set of operations distinct types can be used easily. Distinct types are explained in detail in the Nim manual, we may explain then in more detail in later sections. For now it is enough that we know about their existence.

Sometimes it makes sense to limit the range of numeric variables to only a sub-range. For this Nim uses the range keyword with this notation: range[LowVal .. HighVal]. Values of this type can never be smaller than LowVal or larger than HighVal. For Nim v1.6 we can define range types also by leaving out the range[], that is by just two constants separated by ...

For the above example the base type of the defined ranges is int, so the ranges are compatible with the predefined int type, we can assign values of int type to our range types and vice versa. In our example the size of the range types is the size of the int base type, but of course we could use other base types, like type Weekday = 1.int8 .. 7.int8. If we try to assign to a range type a value that falls not into the allowed range then we get a compile-time or run-time range error. This can help us to prevent or to discover errors in our programs. Note that whenever we use range types, the compiler may have to add additional checks to ensure that variables are always restricted to the specified range. This check is active in debug mode and also when we compile with option -d:release, and is only ignored when we compile with -d:danger or explicitly disable range checks. So using a lot of range types may increase code size and decrease performance. For the example above, the line with the assignment d = a generates a runtime check. An important and often used range type is the data type Natural defined as range[0 .. int.high]. That type is compatible with the int type and does not wraps around as uint would do. It is often used as type for procedure parameters when the arguments has to be not negative. In the proc body we sometimes copy arguments of natural type to an ordinary integer — that way we can ensure a none negative start value, and can avoid many range checks in the procedure body.

We can also declare sub-range types with float base types like type Probability = range[0.0 .. 1.0].

Note that we can still mix different sub-range type:

Such an operation is generally a bug, to prevent it we can put the distinct keyword in front of our ranges. But then again we have to define the allowed operations our self or to borrow them from the base type.

While enums in C are nothing more than integers with some special syntax for creation, Nim’s enums are more complex.

In Nim enums can be used whenever some form of symbols are needed like the colors red, yellow and green of a traffic light or the directions north, south, east and west for a map or a game.

Most of the time we declare an enum type and the corresponding values by simple listing them like

We can use variables of type TrafficLight then like

Enums support assignment, plain tests for (un)-equality and for smaller or greater. Additional the functions succ() and pred() are defined for enums to get the successor or predecessor of an enum, ord() or int() deliver the corresponding integer number and the $ operator can be used to get the name of an enum. We can also iterate over enums, so we can print all the colors of our TrafficLight by

Ordinary enums start at 0 and uses continues numbers for the internal numeric value, so that enums can be used as array indices.[19]

But we can also assign custom numbers like

We should avoid that, as these "enums with holes" generate some problems for the compiler and may be later deprecated. For example array indexing or iterating is obviously not possible for enums with holes.

It is also possible to set the string that the stringify operator $ returns, like in

Here the assigned numerical values should be 0, 2 and 3. Currently the enums numerical values must be specified in ascending order always.

When we have many enums in a program then name conflicts may occur, for example we may have an additional enum type named BaseColor which also has red and green members. For that case the {.pure.} pragma exists:

With the pure pragma applied we can use the full qualified enum name when necessary, like BaseColor.red. But we can still use unqualified names like blue when there is no name conflict.

Boolean types are used to store the result of logic operations. The type is called bool in Nim and can store only two values, false and true. Although we have only two distinct states for a boolean variable and so one single bit would suffice to store a bool, generally a whole byte (8 bits) is used for storing a boolean variable. Most other programming languages including C do the same. The reason is that most CPU’s can not access single bits in the RAM — the smallest entity that can be directly accessed in RAM is a byte. The default initial state of a boolean variable is false, corresponding to a byte with all bits cleared.

In line three we assign to the variable adult the result of a logical comparison. The next two lines assign the boolean constants true and false to the variables, with their type bool inferred.

Variables of type bool support the operators not, and, or and xor. Not inverts the logic value, a and b is only true when both values are true, and false otherwise. And a or b is true when at least one of the values is true, and only false when both values are false. Xor is not used that often. It is called exclusive or, a xor b is false when both values have the same logic state, that is when both are true, or both are false. When the values are not the same, than the result of the xor operator is true. The xor operator makes more sense for bit operations, which we will learn later — for the boolean type, a xor b is identical to a != b.

When using conditional execution, some people like to write expressions like if myBoolExp == false:, which is identical to if not myBoolExp:. Well they may do, but please never write if myBoolExp == true:, that looks really too stupid.

Sometimes it is useful to know that false is mapped to the int value 0, and true to the int value 1. That is similar to the C language, but C has not a bool type, instead the numerical value 0 is interpreted as false in conditional expressions, and all none zero values are interpreted as true.

The effect of the last line is identical to the if statement above. In very, very rare cases working with the actual int value of boolean variables may make sense, but generally we should avoid that. Later in the book there is a section about branchless code where we will present a proc that actually may get faster by using such a trick.

The data type for single characters is called char in Nim. A variable of type char has 8 bits and can store single characters. Indeed it stores 8-bit integers which are mapped to characters. The mapping is described by the ASCII table. For example the integer value 65 in decimal is mapped to the character A. When we use single character literals, then we have to enclose the letter in single quotes. As only 8 bits are used to store characters, we only have 256 different values, including upper and lower case letters, punctuation characters and some characters with a special meaning like a newline character to move the cursor in the terminal to the next line, or a backspace character to move the cursor one position backwards. Single characters are not used that often since we generally group them in sequences called strings to build text.

The initial ASCII table contains only the characters with numbers 0 up to 127, here is an overview generated with the small program listed in the appendix:

The position in the table is the sum of the number on the left and the number on the top, i.e, character A has position 64+1=65, which is the value the Nim standard function ord('A') or int('A') would return. The characters with a decimal value less than 32 can not be printed and are called control characters, like linefeed, carriage return, backspace, audible beep and such. Character 127 is also not printable, and is called DEL. An important property of this table is the fact that decimal digits and upper- and lower-case letters form contiguous blocks. So to test for example if a characters is an uppercase letter we can use this simple condition: c >= 'A' and c <= 'Z'.

Characters with ord() > 127 are so called umlauts, exotic characters of other languages, and some special characters. But these characters may be different on different computers, as the characters depend on the active code-page, which maps position to actual character, and there are multiple code pages. When we need more than the plain ASCII characters, then we use strings in Nim, which display many more glyphs by using UTF-8 encoding.

The control characters with a decimal value less than 32 can not be typed on the keyboard directly and for some characters with decimal value greater than 126 it can be difficult to enter them on some keyboards. For these characters as well as for all other characters escape sequences can be used. Escape sequences start with the backslash character, and the following characters are interpreted in a special way: The backslash can follow a numeric value in decimal or hexadecimal encoding, or a letter which is interpreted in a special way. We mentioned already that the character 'A' is mapped to the decimal value 65, which is its position in the ASCII table. So instead of 'A' we could use the escape sequence '\65' for this character. Or, as decimal 65 is 41 in hexadecimal notation (4 * 16^1 + 1 * 16^0) we can use '\x41' where the x indicates that the following digits are hexadecimal. For common, often used control characters it is not easy to remember their numeric value, so another notation with a letter following the backslash can be used. For the important newline character we can use the decimal numeric value '\10', the hexadecimal value '\xA' or the symbolic form '\n'. Here the letter n stands for newline.

We can regard the backslash character, which introduces escape sequences, as a special hinting symbol for the compiler: Caution, the following characters must be interpreted in a special way.

It is important that you understand that all these escape sequences are only a way to help the programmer to enter these invisible control characters — the compiler replaces the control sequences immediately with the correct 8 bit value from the ASCII table, so in the final compiled executable '\65' or '\n' are both only a plain 8 bit integer value:

The following table lists a few important control characters:

The hexadecimal numbers after the \x character can be upper or lower case and can have one or two hexadecimal digits. For symbolic control characters like '\a' for alert the upper case variant '\A' seems to be identical currently. The single quote entered as ''' does give an error message, so you have to escape it as '\''. Unfortunately by supporting this form of escaping it becomes impossible to enter a backslash character directly, so we have to escape the backslash character as '\\' to print a single backslash.

For Nim the most important control character is '\n' which is used to start the output in a terminal window at the beginning of a new line. But '\n' is generally not used as a single character but embedded in strings, that is sequences of characters. We will learn more about strings soon. Note that the echo() function inserts a newline character automatically after each printed line, but the write() function does not:

What may be a bit confusing is the fact that we use the backslash character as escape symbol, and at the same time above table has an entry '\e' which is also called [ESC]. These '\e' control character with decimal value 27 is fully unrelated to the backslash character that we use to type in control characters. [ESC] is a different special character to start control sequences, it was used in the past to send special commands to printers or modems, and can be used to control font style or colors in terminal windows.

Nim’s control characters should, with few exceptions, be identical with control characters of the C language, so you may also consult C literature for more details.

In Nim integers, enumerations, characters and the boolean types are ordinal types. Ordinal types are countable and ordered, and for each of these types a lowest and largest member exists. The integer ordinal types supports the inc() and dec() operations to get the next larger or next smaller value, and the other ordinal types use succ() and pred() for this operation. These operations can produce overflow or underflow like errors if applied to largest or smallest value. The function ord() can be used on ordinal types to get the corresponding integer value. Note that unsigned integers are currently not called ordinal types in Nim, and that these unsigned types wrap around instead of generation overflow and underflow errors.

From mathematics we know that sets are some form of unordered collection for which we can test membership (x is included in mySet) and we can perform general set operations like union of multiple sets. In Nim we can have sets of all the ordinal types and the unsigned integer types, but due to memory restrictions integer types larger than two bytes can not be used as set base types. All elements in a set must have the same base type. A set can be empty, or it can contain one or multiple elements. For a specific element it can be contained in a given set or it can be not contained, but it can be never contained multiple times. One very basic set operation is the test if an element is contained in a set or is not contained in it. Sets are unordered data types, that is sets containing the same elements are always equal, it does not matter in which sequence we added the elements. Important set operations are building the union and building the difference of two sets with the same base type: The union of set a and set b is a set which contains all the elements that are contained in set a or in set b (or in both). The intersection of set a and set b is a set which contains only elements which are contained in set a and in set b.

The mathematical concept of sets maps well to words and bits of computers, as most CPU’s have instructions to set and clear single bits and to test if a bit is set or unset. And CPU’s can do and, or and xor operations which corresponds to the union and intersection operation in mathematical set.

Nim supports sets with base type bool, enum, char, int8, uint8, int16 and uint16. Note that we need a bit in the computer memory for each member of the base type. The types char, int8 and uint8 are 8 bit types and can have 2^8 = 256 distinct values, so we need 256 bits in the computer memory to represent such a set. That would be 32 bytes or four 64 bit words. To represent a set of the base type uint16 or int16 we need already 2^16 bits, that is 2^13 bytes or 2^10 words on a 64 bit CPU. So it becomes clear that supporting base types with more than 16 bit makes not much sense.

While testing if an element is included or is not included in a set with the in or notin operators is always a very fast operation, other operations like building the intersection or union and set comparison operations may be not that fast when we use the int16 or uint16 base types, as for these operations the whole set, that is 2^10 words on a 64 bit CPU, has to be processed.

We will start our explanation with sets with character base type as these sets are very easy to understand and at the same time very useful. Let us assume that we have a variable of character type and we want to test if that variable is alphanumeric, that is if it is a lower or upper case letter or a digit. A traditional test would be (x >= a and x <=+z) or (x +>= A and x <= Z) or (x >= 0 and x <= 9). Using Nim’s set notation we can write that in a simpler form:

Here we defined a constant of set[char] type which contains lower and upper case letters and the decimal digits. We used the range notation to save us a lot of typing ({'a', 'b', 'c', …​}). It works in this case only, as we know that all the lowercase letters, the upper case letters and the decimal digits built an uninterrupted range in the ASCII table.

With that definition we can use a simple test with the in keyword. These test is equivalent to the procedure call AlphaNum.contains(x). In compiled languages (most) set operations are generally very fast as they map well to CPU instructions.

Older languages like C have not a dedicated set data type, but as sets are so useful and efficient, C emulates these operations by using bit-wise and and or operations in conjunction with bit shifts.

Two important operations for sets are building the union and the intersection:

The constant ANMO would now contain all the characters from AlphaNum and from MathOp, that is letters, digits and math operators. The constant Empty would get all the characters that are at the same time contained in set AlphaNum and in set MathOp. As there is not a single common characters, the set Empty is indeed empty. Remembering the two operators + and * for union and intersection is not easy. For the intersection operator * it may help when we imagine the set members as bits, and we assume that we multiply the bits of both operands bitwise, that is we multiply the set or unset bits at corresponding position each. The resulting bit pattern would get set bits only for positions where both arguments have set bits.

We can use the functions incl() and excl() to add or remove single set members:

The result is a set with letters a, b, c and the characters _ and ?. Note that calling incl() has no effect when the value is already included in the set, and calling excl() has no effect when the value is not contained in the set at all.

Another operation is the difference of two sets — a - b is a set which contains only the elements of a which are not contained in b. In Nim there is currently no operator for the complement or the symmetric difference of sets available. We can produce a set complement by using a fully filled set and then removing the elements of which we want the complement. For a character set that would look like {'\0'..'\255'} - s, where s is the set to complement. And the symmetric difference of set a and set b can be generated by the operation (a+b) - (a*b) or by (a-b) + (b-a).

As the not operator binds more tightly than the in operator, we have to use brackets for the inverted membership test like not(x in a) or we can use the notin operator and write x notin a. We can test for equality of sets a and b like a == b and for subset relation a < b or a <= b. a <= b indicates that b contains at least all members of a, and a < b that b contains all members of a and at least one more element.

Finally we can use the function card() to get the cardinality of a set variable, that is the number of contained members.

We should also mention that we can have character sets which are restricted to a range of characters:

In the code above the compiler detects the assignment to variable y as invalid.

Set of numbers work in principle in the same way as sets of characters. One problem is that in Nim integer numbers are generally 4 or 8 bytes large, but sets can contain only numbers with 1 or 2 byte size. So we have to specify the type of set members explicitly:

In the code above we defined a set type which can contain int8 numbers in the range 0 to 63.

We can use also another notation for numeric sets when we define an explicit range type like in

What may be a bit surprising is the fact that Nim’s sets work also for negative numbers:

Enum sets are also very useful and can be used to represent multiple boolean properties in a single set variable instead of using multiple boolean variables for this purpose:

Enum sets can be used to interact with functions of C libraries where for flag variables often or’ed ints are used. For example for the gintro C bindings there is this definition:

Here the {.size.} pragma is used to ensure that the byte size of that set type matches the size of integers in C languages.

When we define set of enums in this way to generate bindings to C libraries, then we have to ensure that the enum values start with zero, otherwise Nim’s definition will not match with the C definition. For example in the gdk.nim module we have

The first enum with ordinal value zero was automatically added by the bindings generator script to ensure type matching. Nim devs sometimes recommend to use plain (distinct) integer constants for C enums. That may be easier, but integer constants provide no name spaces, so names may be aFlagWheel instead of AxisFlag.wheel or plain wheel when there is no name conflict for pure enums. And with integer constants we have to combine flags by or operation like (aFlagWheel or aFlagSlider) instead of clean {AxisFlag.wheel, slider}.

Can we print sets easily? As sets are an unordered type, it is not fully trivial, but we can iterate over the full base type and check if the element is contained in our set like

We will learn how the for loop work soon. Note that the sequence in which the set members are printed is determined by our query loop, not by the set content itself, as sets are unordered types.

The string data type is a sequence of characters. It is used whenever a textual input or output operation is performed. Usually it is a sequence of ASCII-only characters, but multiple characters in the string can be interpreted as so called utf-8 unicode characters, that allow displaying nearly unlimited symbols as long as all the needed fonts are installed on your computer and you manage to enter them — unicode characters may be not accessible by a simple keystroke. For now we will only use ASCII characters, as they are simpler and work everywhere. String literals must be enclosed in double quotes. Nim strings are similar to the Nim seq data types: both are homogeneous variable-size containers. That means that a string or a seq expands automatically when you append or insert characters or other strings. Don’t confuse short strings with only one character with single characters: A string is a non trivial entity with internal state like data buffer (the actual contained characters), length and storage capacity why a variable of char type is nothing more than a single byte interpreted in a special way. So a string like "x" is fully different from 'x'.

In the above example code we declare a variable called str and assign it the initial literal value "Hello". We use the echo() procedure to ask the user for his name, and use the readLine() procedure to read the user input from the terminal. To show how we can add characters to existing string variables we call the add() procedure to append a space character to our str variable, and finally call the echo() procedure to print the hello message and the name to the screen. Note that the echo() procedure automatically terminates each output operation with a jump to the next line. If you want an output operation without a newline, you can use the similar write() procedure. But write() needs an additional first parameter, for which we use the special variable stdout when we want to write to the terminal window.

So we could substitute the last two lines of the above code by

The Nim standard library provides a lot of functions for creating and modifying strings, most of these functions are collected in the system and in the strutils module. The most important procedures for strings are len() and high(). The len() procedure returns the length of a string, that is the number of ASCII characters or bytes that the string currently contains. The empty string "" has length zero. Note that the plain len() function returns the number of 8-bit characters, not the number of unicode glyphs when the string should be interpreted as unicode text. To determine the number of glyphs of unicode strings you should use some of the unicode modules. The high() function is very similar to the len() function, it returns the index of the last character in the string. For each string s high(s) == len(s) -1, so high("") is -1. Remember that Nim supports method call syntax, so we can also write s.len instead of len(s).

The most important operators for strings are the subscript operator [] which allows access to individual characters of strings, and the .. slice operator which allows access to sub-strings. The first character in a string has always the index zero. For concatenation of string literals or string variables Nim uses the & operator.

In the example above we define the string variable s by use of two literal strings to show the use of the concatenation operator. In line two we use the slice operator to replace the sub-string "hate", that is the characters with index position 3 up to 6, with the string literal "like". In this case the replacement has exactly that many characters as the text to replace, but that is not necessary: We can replace sub-strings with longer or shorter strings, which includes the empty string "" to delete a text area. In the last line of above example we use the subscript operator [] to replace the single character '?' at the end of our string with an exclamation mark. For subscript and slice operators Nim supports also a special notation which indicates indexing from the end of the string. Python and Ruby use negative integers for this purpose, while Nim uses the ^ character. So [^1] is the last character, [^2] the one before the last. So we could have written s[^1] = '!' for the last line of our code fragment above. The reason that Nim does not use negative integers for this purpose is that Nim arrays don’t have to start at index zero, but can start with an arbitrary index including negative indices, so for negative indices it may be not always clear if a regular index or a position from the end of the string is desired. The term s[^x] is equivalent to s[s.len - x]. We will learn some more details about the slice operator in a later section when we have introduced arrays and sequences.

Another important operator for strings is the "toString" or stringify operator $. It can be applied to variables of nearly all data types and returns its string representation which can then be printed. Some procedures like echo() apply this operator on its arguments automatically. When we define our own data types then it can make some sense to define the $ for them, in case we need a textual representation of our data — maybe only for debugging purpose. Note that applying the $ operator on a string directly has no effect and is ignored, as the result would not change.

Strings can contain all characters of the char data type including the control characters. The most important control character for strings is the newline character '\n' which is used at the end or sometimes also in the middle of strings to start a new line. For strings Nim also supports the virtual character "\p" to encode an OS dependent line break. When compiled for Windows, "\p" is automatically converted to "\r\n", and to a plain '\n' on Linux. Note that "\p" can be used in strings, but not as single character, as it is two byte on Windows. "\p" is only needed to support very old Windows version or maybe other exotic operating system, as modern Windows recognizes plain '\n' well.

As strings support utf-8 unicode, an escape sequence starting with "\u" is supported to insert unicode codepoints. The "\u" follows exactly 4 hexadecimal digits or an arbitrary number of hex digits enclosed in curly braces {}.

As string literals are enclosed in quotation marks, it follows that strings can not directly contain this character, we have to escape it as in "\"Hello\", she said".

Maybe we should mention that Nim strings use copy semantics for assignment. As we have not yet introduced references or pointers, copy semantics is what you should expect — strings behave just like all the other simple data types we used before like integer or float numbers or enums and characters:

The output is

The assignment s2 = s1 creates a copy of s1, so the subsequent add() operation does only modify s1 but not s2. Probably not surprising for you, but other programming languages may behave differently, i.e. the assignment may not copy the textual content but create only a reference to the first string, so that modifying one of then also effect the other. We will learn more about the concept of references when we introduce the object data type.

Entering Unicode Characters

echo "\xe2\x99\x9A \xe2\x99\x94"
echo "\u265A \u2654"
echo "\u{265A} \u{2654}" # {} is only necessary for more than 4 hex digits
echo "♚ ♔"

The CString data type

Escape Sequences in Strings

echo "\n"
echo "Hello\nHello\nHello"

Raw Strings and multi-line Strings

echo r"\n"
echo "\\n"

echo """this is
three lines
of text"""

echo "this is\nthree lines\nof text"

Comments are not really a data type, but they are also important. Ordinary comments starts with the hashtag character # and extend to the end of the line. The \# character itself and all following characters up to the line end are ignored by the compiler. You can also start the comment with \\##, then it is a documentation comment. It is also ignored by the compiler, but can be processed when you use later tools to generate documentation for your code. Documentation comments are only allowed at certain places, often they are inserted at the beginning of a procedure body to explain its use. There are also multi-line comments, which starts with the two characters #[ and ends with ]#. These form of comment can extend over multiple lines and can be nested, that is multi-line comments can contain again plain or multi-line comments.

Multi-line documentation comments also exists and can also be nested.

You can also use the #[ comment ]# notation to insert comments everywhere in the source code at places where a white-space character is allowed, but these form of in source comments is rarely used.

There exists some more predefined types like the container types array and seq, which can contain multiple elements of the same type, or the tuple and the object type which can contain data of different types. Nim tuples and objects are similar to C structs, they are not so verbose as Java classes. We will learn more about all these types in later sections of the book.

var
  pos2: int # OK
  leftMargin: int # OK
  next_right_margin: int # OK
  _privat: int # illegal
  custom_: int # illegal
  strange__error: int # illegal

Like most other programming languages, Nim has the concept of code blocks or scopes. The body of procedures, functions, iterators and templates, as well as the body of various loop constructs or code following conditional statements builds an indented block and creates a new scope. In this new scope we can define variables, named constants, or types with the var, let, const and type keywords which are local to this block. These symbols are only visible in this scope, and local variables that need storage are actually created when the program executes the block, and destroyed when the block is left. Well, in principle, and at least for ordinary stack allocated value variables, for references and pointer variables, things are a bit more complicated, we will discuss that in more detail when we introduce references. Here we have used the term code block, to clearly separate them from the const, var, type and import sections which are a different form of indented blocks. Remember that the compiler processes our program code from the top to the bottom, so we have always to define symbols before we can actually use them. When we define an entity in a code block, and a symbol with that name was already declared before outside of this block, then that symbol is shadowed, that is the prior declaration gets temporary invisible.

While we have not officially introduced procedures as units to structure our program code yet, we have put the above code this time by intend into the body of a proc called doSomething().

This way we can guarantee that the two variables a and b defined in that proc are really stack allocated. Actually in real life programs nearly all of the program code is embedded in procs. We will discuss the peculiarity of global code later. In the proc body from above the two variables a and b are local to the proc doSomething() — they are created on the stack when the procedure is called, that is when we ask to start it execution by a statement like doSomething(). These two variables are never visible in code outside of this proc, and the storage for these two variables is automatically released when execution of that procedure ends, in this case when the last line of the proc is reached. In the body of the proc we define although a new custom type and a named constant — just to show that it is possible. Both symbols are also local to this proc and invisible outside.

The indented block following the if b: statement is sometimes called a "if then" block or just if block — in that block we define two other variables called a and sum of float type, which are also stack allocated. If these two variables are already allocated when the proc starts its execution, or only when the then block following the if statements is executed, is actually an implementation detail. As the variable a of float type in the if then block has the same name as the outer variable of int type, that integer variable is shadowed in the if block — the outer value gets temporary invisible as soon as the new symbol is declared. Other symbols of outer scopes remain visible. In the if then block as well as in most other indented code blocks we could also define named constants or custom types, these would be visible only in this block. Indented code blocks can be nested — in one block we can have more indented block, for which all declared symbols are again local and invisible outside. The last echo() statement in our code example from above is already below the if then block, so the initial variable a of integer type becomes again visible.

In the introducing sections of the book we have generally used program code at a global level, not embedded in a proc body. We did that for simplicity and as we had not already introduced procs. Global code is sometimes used in small scripts or for special purposes, like program initialization. But for larger programs most of the code is generally grouped in procedures. For variables defined in global code it is not that well defined where they are stored, it may depend on the actual Nim compiler implementation and the compiler backend. The performance of global code can be worse than code enclosed in proc bodies, so when performance maters we should put our code in procs. One reason for the not optimal performance of global code is, that global variables are not located on the stack,but in the global BSS segment of the program, and that the backend can not optimize global code that well, e.g. global variables may not be cached in CPU registers. Note that variables that have to exists and keep it value for the whole runtime of the program, and not only for the duration of the execution of a single proc, has to be defined as globals. The same holds obviously for global variables that are used from code of different procs, like the stdout and stdin variables of the system module. An alternative to the use of global variables when a variable used in a proc should keep its value between different proc calls is to attach the {.global.} pragma to a proc local variable. This way that variable is still only visible in that proc where the variable is declared, but the variable is stored in the BSS segment instead on the stack and so its value is preserved between proc calls.

Note that structured named constants, e.g. constant strings, are stored also in the BSS segment, even when they are only defined local to a proc. So large structured constants can increase the executable size, as the BSS segment is a part of the program executable.

The space character with decimal ASCII value 32 is used in Nim program code to indent code blocks and to separate different symbols from each other. Nim keywords are always separated with leading and trailing white-space from other symbols, while other symbols are most often separated by punctuation and an additional optional space character. Whenever the syntax allows a space, we can also insert multiple spaces or a comment enclosed in #[ ]# in the source code. Tabulator characters are not allowed in the Nim source code, but we can use them in comments and of course in string literals. We mentioned already, that spaces can make a difference how operators or function parameters are handled. In expressions like a+b or a + b the + operator is regarded as an infix operator, but in a + -b the minus sign is regarded as unary operator bound to b. This way asymmetric expressions like a +b or a <b would be invalid, as the operators are interpreted as unary ones attached to b, and then there is no infix operator between the two variables. A proc call like echo(1, 2) is interpreted as a call of echo() with two integer literal arguments, while a call like echo (1, 2) with a space after the proc name is interpreted in command invocation syntax as a call with a tuple argument. While in C code it is not uncommon to always insert a space between the function name and it parameter list, we should avoid that in Nim for the described reason. We will learn more about proc calls and the tuple data type later.

Nim uses the following punctuation characters as operators:

These symbols can be used as single entities or in combination, and we can define our own operators or redefine existing operators. All these symbols can be used as infix operators between two arguments, or as unary prefix operators, but Nim does not support unary postfix operators, so a notation like i++ as known from the C language is not possible in Nim. There exists a few combinations of these punctuation characters that have a special meaning, we will learn more about that and how we can define our own operators later in the book.

In Nim these keywords are also used as operators:

Operators have different priorities, e.g. * and / have a higher priority than + and -. In most cases the priority is just as we would expect, maybe with a few exceptions. If we are unsure, we can group terms with brackets, or consult the compiler manual for details.

Since version 1.6 Nim also allows to define and use a few unicode operators, but these are still considered experimental. For details you should consult the compiler manual.

Global program code or code in called procs is generally executed from top to the bottom and from left to the right, as long as control structures do not enforce a different order. To demonstrate this, we use here a set of four different procs, which contain an echo() statement each, and return a numeric expression. Well, we have not yet formally introduced procedures, so if you should feel too uncomfortable with the code below, just skip this section for now and come back when you have read the section about procs:

It should be no real surprise, that the first three echo() statements produce this output:

For the term d(c(3)) it is obvious that the inner expression c(3) has to be evaluated first, before that result can be used to call proc d().

The last two lines demonstrate the so called short-cut-evaluation for expressions with the boolean and or or operators. As the expression a() and b() is always false when a() is false, for this case b() has not to be evaluated at all. In a similar way, as the expression a() or b() is always true when a() is true, for that case b() has not to be evaluated at all. So in the last two lines of above code b() is never called at all, and the output is just

Note that in Nim as in most other programming languages the assignment operator = behaves different compared to ordinary operators like + or *, as in assignments like let a = b + c() obviously the right side has to be evaluated before the result can be actually assigned to variable a.

The if statement with multiple optional elif branches and an optional else branch evaluates a sequence of boolean conditions at program runtime. As soon as one condition evaluates as true the corresponding statement block is executed, and after that the program execution continues after the whole if construct. That is at most one branch is executed. If none of the conditions after the if or elif keywords evaluates to true, then the else branch is executed if it exists. A complete if statement consists of one if condition, an arbitrary number of elif conditions and one optional else part:

The most simple form of an if statement is

Note that the branches are indented by spaces, we use two spaces generally, but other numbers work as well. And note that it is elif, not elsif like in Ruby, and that there is a colon after the condition. Instead of a single statement we can use multiple in each branch, all on its own line and all indented in the same way.

When there is no elif and no else part, then we can also write the conditional code direct after the colon, like

With an elif and an else branch the example from above may look like

Note that we perform the age tests in ascending order — it would not make much sense to first test for a condition age < 6, and later to test for age < 4, as the if statement is evaluated from top to bottom, and as soon as one condition is evaluated as true, that branch is executed and then the program execution continues after the whole if construct. So a later test age < 4 would be useless, when that condition is already covered by a prior test age < 6.

As the various conditions of the if statement are processed from top to bottom until one condition evaluates to true, it can be a good idea to put the most likely conditions first for optional performance, as then the unlikely conditions have not to be evaluated in most cases. Another strategy for larger if/elif constructs is to put the most simple and fast tests to the top when possible.

We can also have if/else expressions which returns a value like in

In C for a similar construct the ternary ? operator is used.

In languages like C or Ruby the assignment operator "=" is an expression which returns the assigned value, so in C we can write code like

In Nim the assignment operator is not an expression with a result, but we can group multiple statements in round brackets separated by semicolon, and when the last statement in the bracket is an expression, than the whole bracket has the same value. So we can use conditional terms like

If we declare a variable in this way using the var or let keyword then that variable is only visible in the bracket expression itself and in the following indented block.

Note that if-expressions must always return a well defined value, so they must always contain an else branch. A plain if, without an else, or an if/elif without an else does not work. And as Nim is a statically typed language and all variables have a strictly well defined type, the if-expression must return the same type for all branches!

The when statement is syntactically very similar to the if statement, but while all the boolean conditions are evaluated during the program run time for the if statement, for the when construct all the when/elif/else conditions have to be constant expressions and are already evaluated at compile time. In ordinary program code the when statement is not used that often, but it is useful when we write bindings to C libraries and low level code. Common use cases for the when statement are the isMainModule condition test and the test for defined symbols like defined(windows):

The value isMainModule is only true for a source code file, when that file is compiled directly as main module, that is when it is not indirectly compiled because it is imported by other modules. This way we can include easily test code to our library modules — that test code is ignored when the module is used as library, but active when we compile the module direct for testing.

A when defined() construct can be used to test for predefined or our own custom options, e.g. we may give the optional option -d:gintroDebug to the compiler and test in the code of that module for this option, like when defined(gintroDebug):.

One difference of the when to the if statement is, that the "then" branches do not open a new scope, so variables which we define there are still visible after the construct has been processed:

Another peculiarity of the when statement is, that it can be used inside object definitions — we will show an example for that in a later section of the book when we introduce the object data type. In the same way as the if construct, when can also be used as an expression.

The case statement is not used that often, but it can be useful when we have many similar conditions:

To enable optimizations the case construct has some restrictions compared to a more flexible if/elif statement:

The variable after the case keyword must have a so called ordinal type like int, char or string, while float would not work. And the values after each of keyword must be constant, that is a single constant value, multiple constant values or a constant range like 'a' .. 'd' for the 4 first lower case letters. Of course these constants must have a type compatible to the type of the variable after the case keyword. A case statement must cover all possible cases, so most of the time an else branch is necessary.

For Nim version 1.6 the case statement can contain also optional elif branches with arbitrary boolean conditions. This was not the case in the wirthian languages Pascal, Modula and Oberon, and makes Nim’s case construct now very similar to the ordinary if/elif/else.

Unless the similar switch statement in C the case statement needs no break after each branch. If a condition after a of keyword is true, then the corresponding statement or statement sequence is executed, and after that program execution continues after the whole case construct.

The case construct can also be used as an expression like in

Here an else is necessary to cover all cases. And as you see we can also indent the block after the case keyword if we want.

The while loop is used when we want to do conditional repetitions, that is, if we want to check a condition and want to execute a block of statements only as long, as the condition is true. If the condition is false in advance, or becomes false after some repetitions, then the program execution proceeds after the indented loop body block.

A basic while loop has this shape:

That loop would print the message three times. Like the condition in the if clause the condition is terminated with a colon. Note that the condition must change during execution of the loop, otherwise, when the condition is true for the first iteration, it would remain true and the loop would never terminate. We decrease the loop counter repetitions in the loop, so at some point the condition will become false and the loop will terminate and program execution will continue with the first statement after the loop body. Note how we decrement the loop counter: The right site of the assignment operator is evaluated, after that is done, the new value is assigned to the counter.

There exists two rarely used variants of a while loop: the loop body can contain a break or a continue statement, which each consists only of this single keyword. A break in the body stops execution of the loop immediately and continues execution after the loop body. And a continue statement in the body skips the following statements in the body and starts at the top again, the while condition is evaluated again.

Above code used the == and the != operators. The == operator does a test for equality, and != test for inequality. Both operator work for most data types like integer, floats, characters and strings. The literal value of an empty string is written "". In line 2 we test if the variable named input has not the value "quit", and in line 4 we test of that variable is empty, that it contains no text at all.

Using of break and continue destroys the expected flow in loops, it can make understanding loops harder. So we generally avoid their use, but sometimes break or continue are really helpful. For example when an unexpected error occurs, maybe by invalid user input.

There in no repeat loop as in Pascal in Nim, which does the first check at the end of the loop when it was executed already for the first time. Repeat loops are not used that much in Pascal, and they are some sort of dangerous, because they check the condition after the first execution of the body, so maybe the body is executed with invalid data for the first iteration. Later, we will see how we can use Nim macros to extend Nim by a repeat loop that can be used as it would be part of Nim core functionality.

The block statement can be used to create a new indented block, which creates a new scope, in the same way as a when true: statement would do:

Blocks can be useful to structure large code segments, when there are no better ways, as splitting the code in multiple procs, available. For testing purposes blocks can be useful too, to keep symbol in a local scope. But actually most useful are blocks, when the blocks get attached names, and we use the break statement in a while or for loop to break out of a nested loop:

The break check statement would immediately leave the nested loops and continue with the first statement after the block, which is the last line in the code segment above. Using break in such a way is not very nice, as it may make it harder to understand the code structure, but sometimes it can be very useful.

These are very useful and important in Nim and other programming languages. For loops are most often used to iterate over containers or collections. We have not discussed the important array and seq containers yet, but we know already the string container.

A string contains characters, the characters are numbered starting with 0, and we can access single characters of a string with the subscript operator [], which gets the position of the desired character as argument. So we could print the single characters of a string, in this way:

It is obvious that the pos variable is some sort of annoying here — we want to process all the characters in the string in sequence, so why would we have to use a position variable to do that. And this way is susceptible to errors, maybe we forget increasing the pos variable in the loop body. So most modern languages provide us with iterators for this purpose:

That is obvious shorter. The for construct may appear a bit strange, and it is indeed, but it is a common way to write iterators, it is used in Python too. Ruby uses something like s.each{|ch| …​} instead.

For loops in Nim iterates over containers or collections, and pics each element in sequence in this process. The variable after the for keyword is used to access or to reference the single elements. That variable has automatically the right type, which is the type of the elements in the container, and get in each iteration the value of the next element in the container, starting by the first element in the container and stopping when there is no element left. Items() is here the actual iterator, which allows us to access the individual characters in sequence. It exists the convention in Nim that an items() iterator is automatically called in a for loop construct when no iterator name is explicitly given, so we could also write shorter for ch in s: in this use case.

You may recognize that the output of the above for loop is not identical to the output of the previous while loop. The while loop stops when the last character, that is '?', is reached, while the for loop processes this last character still. That is intended for the for loop, its general purpose is to process all the elements in containers or collections.

The above for loop does a read access to the string, that is, we get basically a copy of each character, and we can not modify the actual string in this way. When we want to modify the string, there is a variant available.

Here we use mitems() instead of the plain items(), the leading "m" stands for mutable. In the loop body we can assign different values to the actual content.

type
  Computer = object
    manufacturer: string
    cpu: string
    powerConsumption: float
    ram: int # GB
    ssd: int # GB
    quantity: int
    price: float

var
  computer: Computer

computer.manufacturer = "bananas"
computer.cpu = "x7"
computer.powerConsumption = 17
computer.ram = 32
computer.ssd = 1024
computer.quantity = 3
computer.price = 499.99

import std/with
var
  computer: Computer
with computer:
	manufacturer = "bananas"
	cpu = "x7"
	powerConsumption = 17
	ram = 32
	ssd = 1024
	quantity = 3
	price = 499.99

computer.quantity = computer.quantity - 1 # we sold one piece
echo computer.quantity

var
  a: array[8, int]
  v = 1
for el in mitems(a):
  el = v
  inc(v)
for el in mitems(a)
  el = el * el
for square in a:
  echo square

var
  s: seq[int]
  v = 0
while v < 8:
  inc(v)
  add(s, v)
for el in mitems(s)
  el = el * el
for square in s:
  echo square

var s: seq[int] = newSeq[int](8)
var i: int
while i < 8:
  s[i] = i * i
  inc(i)

Let us investigate at the end of this section some internal details about arrays and sequences. Beginners not yet familiar with the concept of pointers should better skip this subsection, and maybe come back later. We could consult the Nim language manual or the compiler source code to learn more details about arrays and sequences. Or we can write some code to test properties and behavior. Let us start investigating an array:

When we run this program we get this output:

The size of the whole array is 32, as we have 4 elements each 8 byte in size. And the address of the array itself as well as the address of its first element is identical. Remember that the actual address values will be different for each run of our program, and they may be totally different on different computers, as it is some random choice of the OS which free memory area is used to run our program. This result is expected as the array is a plain block of memory stored on the stack. And indeed the array has copy semantic, when we create a copy called a2 and later modify a, then the content of a2 is unchanged. That was not really surprising, so let us investigate a sequence:

When we run above code we get:

The first two lines of the output may confuse us, as a size of only 8 byte may indicate a plain pointer value on a 64 bit system. Indeed the sequence is not a large object that contains size and capacity fields, but only a tiny object that contains a single pointer to the data storage of that sequence. We know that it is not a plain pointer or ref by the fact that we can not assign nil or test for nil for sequences. (But an object which contains only a pointer is basically identically to a plain pointer, as Nim objects have no overhead as long as we do not use inheritance and when no padding to word size is needed for tiny fields like int8.) Capacity and length are stored also in the memory block that is allocated for the elements as long as the sequence is not empty. So empty sequences don’t wast much memory when we have a lot of them, i.e. arrays or sequences of sequences (matrices). We use the dummy int variable in the code above as we know that plain ints are stored on the stack, and when we compare the addresses of our dummy variable and our sequence, then we see that the addresses indicate close neighborhood, so the seq object is also stored on the stack. But the address of s[0] is very different, indicating that the data buffer is stored in a different memory region, which is the heap. If we would continuously add elements to the seq, then the address s[0] would change at some point, while the address of s would remain always unchanged. That is because the capacity of the data buffer would become exhausted at some point and a new data buffer with a different address would be used. Finally we see again that the sequence has also copy semantic, as the content of the copy s2 remains unchanged when we modify the initial sequence s. We could try to discover some more details of the internals of Nim’s sequences, i.e. we could try to detect where the capacity and size is stored. But that are internal details which should not really interest us and which may change with new compiler version or different compilers.

But OK, you may still not believe what we said, so let us go one layer deeper. We strongly assume that a seq needs a length and a capacity field. And we assume that its data type should be int. We said that both fields should be adjacent to the buffer of the seq elements, that means at the start or at the end. Obviously we can not access the end as long as we do not know the capacity, so capacity field should be at the start, and then length field also. We may find out which one is which by observing the content when seq grows. So let us write some code:

Output when we run the program is:

Don’t worry when you do not understand the program and the output yet. You will better understand it when you have read the sections about references, pointers and memory management. The first two output lines shows us that an uninitialized seq is just a pointer pointing to nil. And the remaining output lines show us the address of the first seq element, the capacity and the length of the seq whenever we add an element. We started with a seq with initial capacity of 4, so address and capacity is constant while we add the first 4 elements. Then the capacity of the allocated buffer is exhausted. A new buffer with different address and doubled capacity is allocated, the already contained elements are silently copied to the start of the new buffer and so on.

type
  HSlice*[T, U] = object   ## "Heterogeneous" slice type.
    a*: T                  ## The lower bound (inclusive).
    b*: U                  ## The upper bound (inclusive).
  Slice*[T] = HSlice[T, T] ## An alias for `HSlice[T, T]`.

proc contains*[U, V, W](s: HSlice[U, V], value: W): bool {.noSideEffect, inline.} =
  result = s.a <= value and value <= s.b

var
  m = "Nim programming is difficult."
m[19 .. 28] = "not easy."
echo m
echo "Indeed " & m[0 .. 18] & "is much fun!"

type
  O = object
    i: int

proc main =
  var
    s = newSeq[O](1000000)
  for i in 0 .. (1000000 - 1):
    s[i] = O(i: i)

  var sum = 0
  for x in s[1 .. ^1]:
    sum += x.i

  for x in items(s.toOpenArray(1, s.high)):

iterator span*[T](a: openArray[T]; j, k: Natural): T {.inline.} =
  assert k < a.len
  var i: int = j
  while i <= k:
    yield a[i]
    inc(i)

for x in s.span(1, s.high):

proc sum(x: openArray[O]): int =
  for el in x:
    inc(result, el.i)

echo sum(s.toOpenArray(1, s.high))

var i, j: int
i = 7
j = i
i = 3
echo i, j

In Nim references are some form of smart or managed pointers, we will learn more about references later. The plain pointer data type is nothing more than a memory address, it is similar to a (unsigned) integer number. We say that a pointer points to an entity when the pointer contains the memory address of that entity.

Beside the pointer data type, which is only some RAM address, we have also the ptr entity. Ptr is not a datatype for its own, it is always used in conjunction with another data type:

Here the variable p is of type pointer, we could use it to point to some arbitrary memory address. The variable ip is of type ptr int, which indicates that it should only point to memory addresses where a variable with data type int resides. So a ptr is a pointer that is bound to a specific data type. Generally we speak only about pointers, if we are referring to an untyped pointer or a typed ptr is generally clear from the context.

When we only declare pointers but do not assign a value then the pointers have the value nil, what indicates that they are regarded to point to nothing. Exactly speaking a pointer can never point to nothing in the same way as an integer variable can not contain no number. As an integer variable always contains a bit pattern, a pointer also always contains a bit pattern. But we are free to define a special pattern as nil, and whenever a pointer has this special value, then we know that it does not really point to something useful. In C instead of nil NULL was chosen for the same purpose. In practice nil and NULL are generally mapped to 0, that is a word with all bits cleared. But that is more or less an arbitrary decision.

So how can we give our pointers above a useful value?

One possibility would be to use Nim’s addr() function, which gives us the memory address of each ordinary variable.

First we declare an ordinary integer variable called number which will reside somewhere in memory when we execute the program, and then we use the addr() function to assign the address of that variable to p and ip. The addr() function is a low level function provided by the compiler, it can be used to determine the memory address of variables and some other entities known to the compiler.[28] We used the echo() procedure to show us the numeric decimal value of the addresses in the terminal. As it generally makes not too much sense to print addresses, echo() would refuse to print it, so we have used the construct cast[int](someValue) to tell that echo() should regard our pointers as plain integer and print it. That operation is called casting, we generally should avoid it, as it destroys type safety, but for learning purposes it is OK to use it. We will learn more about casts and related type conversion later.

The first two echo statements should print the decimal value 0, as the pointers have the initial default value nil.

The echo()s in the last two lines should print a value different from 0, as we have assigned the valid address of an ordinary variable that resides somewhere in the RAM when the program is executed. Both outputs should be identical, as we have assigned for both pointers addr(number) each.

Maybe a funny fact is, that when you run the program multiple times the output of the last two echo() statements print different values. But that is not really surprising — whenever you launch the program, then for our variable number a storage location in RAM is reserved. And that location can differ for each new program execution. For your next holiday in the same hotel, you may get a different room also.

So when we have the pointer ip pointing to a valid address, can we recover the content of that memory region? Sure, we use the de-reference operator [] for that purpose. Whenever we have a typed pointer x we can use x[] to get the content of the memory location where the pointer is pointing to. Note that the operator [] is not really related to the subscript operator [pos] which we used earlier for array, seq and string access. Nim uses ASCII characters for its operators, and that set is not very large. And maybe it would even be confusing when we would have a different symbol for each operator. We can consider [] as some form of content access operator — mystring[pos] gives us the character at that position, and ip[] gives us the content of the memory location where ip points to.

What do you expect as output for the last echo() statement? Note that for the last echo() statement we do not need a cast, as ip[] has a well defined type: ip has type ptr int, so ip[] is of well defined type int and echo() can print the content.

Now let us investigate how we can use pointers to modify the content of variables:

What do you expect for the output of the last echo() statement? Well remember, ip points to the location where variable number is stored in RAM. So echo ip[] gave us the content of number. Now ip[] = 3 is an assignment, the right site of the assignment operator is the literal number 3, which is a value type. Earlier we said that for value types an assignment is a copy operation, the right site of the assignment operator is copied into the variable on the left site. Now ip[] stands exactly for the same content as the name number, and so assigning to ip[] is the same as assigning to number.

In low level programming languages pointer arithmetic can be useful. For example old C code often iterates with pointer arithmetic over arrays by use of constructs like sum += *(myIntPtr++). This was done to maximize performance. Modern C compiler generally understands statements like sum += el[i]; i++ well and generates very good assembly instructions for it. So pointer arithmetic is not necessary in C that often today.

Nim does not provide math operations for pointers directly, but we can always cast pointers to integers and do arbitrary math. And of course we could define our own operators for that purpose, but generally we should avoid that, as it is dangerous, error prone and generally not necessary. As an example let us sum up some array elements:

When we do pointer arithmetic or similar math to calculate the address of variables in the computer memory, then memory addresses are used like integer numbers, and so it makes same sense that Nim’s integers have the same byte size as pointers.

References:

• https://github.com/kaushalmodi/ptr_math

In the previous section we learned the basics about pointers. We used the addr() operator to initialize the pointer by assigning the address of an already existing object. This is in practice not that often done, and it can be a bit dangerous, as it is not always guaranteed that the variable on which we applied addr() will exist as long as our pointer exist. So the pointer may point later to a memory location that is already freed or used by a totally different object already. So the use of addr() is more reserved for advanced programmers who know well what they do, and most of the time addr() is not necessary at all or is only necessary for really low level code, maybe when interfacing with external libraries written in C. Instead of using addr() to assigning to pointers a valid address, often procedures like alloc() or create() are used to reserve a block of memory:

Here the procedure create() is used to reserve a block of memory, the int parameter ensures that the block has the size of an integer value. After ip has a valid value, we can store a value in that memory location and read it again. Note that multiple pointers can point to the same memory location: We declared one more int ptr called ip2. But for that pointer we do not allocate a new block, but we assign the old block that we allocated for ip to ip2. Now both pointers points to the same object, the int value 13. We may call ip2 an alias, as it is a different way to access the same object.

When we use alloc() or create() to allocate memory blocks, then we have to deallocate them when we need them not any more. Otherwise that memory blocks couldn’t be reused. If we would continuously allocate memory blocks and never deallocate, that is free them, then at some point in time all memory would be occupied — not only for our own program, but for all programs running currently on the same computer. We had to terminate our program — when a program is terminated then all resources get freed automatically by the OS.

The use of procedure pairs like alloc() and dealloc() is common practice in low level programming languages like C, but it is inconvenient and dangerous: We can forget to call dealloc() and waste resources, or we may even deallocate memory blocks but still use it by our pointers. The later would at some point of time crash our program, as we would use memory blocks which are already released and may be used for other variables — from our own program or from other programs. Note that in the source code above there is only one single dealloc() call. The reason for that is, that we only allocated one single memory block in one single create() call, ip2 is only one more pointer that points to that block. If we would have used an additional dealloc(ip2) call, then that would be a so called double free error.

As you see, using pointers is inconvenient and dangerous. But still there are situations where plain value type variables do not suffice. The solution of many higher level programming languages to this problem is a Garbage-Collector (GC). The GC does the dangerous and inconvenient task of deallocating unused memory blocks for us automatically.

To distinct the GC managed "pointers" cleanly from the manually managed ones, we call them in Nim references, in some other languages they are called traced pointers. References are always typed like ptr, there is no equivalent to the untyped pointer type for references.

For References we have still to do the allocation our self, then we can use the references, and when we are not using them any more, then the GC frees the corresponding memory block. A typical scenario is that we use references in a procedure or in an otherwise limited block of code: We declare the reference in that code block, allocated and use it, and when the code block is left the GC frees the allocated memory for us. You may think that the fact that we still have to allocate the memory for our references our self is still a problem, as we may forget that step. Well it is not that dangerous, when we forget the allocation step, we would use a reference with value nil, which would immediate result in a runtime error. So we would see the problem immediately. Other pointer errors, like missing de-allocation or use after free are not that obvious and more dangerous.

With references we can rewrite our previous example code in this way:

We have replaced ptr by ref, and instead of alloc() or create() we are using the new() proc which gets the uninitialized ref as a parameter and allocates a managed memory block for it, that is after the new() call ip has a well defined value referring to a managed memory block that can store an integer value. Again, we can use one more ref and assign that ref the value of the other, so now both references the same memory block. The advantage here is that we don’t have to care about freeing that block, the GC will do that when appropriate.

To verify that in the example code above both references really reference the same object in memory, we could add two more lines of code:

Here we are using the reference ip2 to assign to the memory block the literal value 7. After that assignment both echo statements would display that new content.

Using references and pointers to store basic data types like integers is not done that often, in most cases we work with larger objects, and we create some relations between the objects. We will try that in the next section.

You should still wonder for what references are really useful — they seem to be only a more complicated version of plain value type variables.

Now let us assume we want to create a list of things or persons, maybe a list of our previously used Computer data type, or maybe a list of persons we will invite to our next party. We will create the party list for now, as the Computer data type we used before has already many fields, and filling all the fields would be some effort, so let us use a new Friend data type which should store only the friends name for the beginning — we may add more fields later when necessary. So we may have

With that declaration we could declare a few friends variables like

But that is not what we want, we would need a list with all of our friends that we would like to invite to our party, we would want to add friends to the list, and maybe we would want to delete friends also. You may think we could use Nim’s sequence data type for that, and you are right. But let us assume we could not use that predefined Nim data type for some reason. Then we could create a list of linked references to Person.

Now our Friend data type is a reference to an object , and the object itself has an additional next field which is again of type Friend.

That is some sort of recursion. If that should appear as too strange, then imagine you have some numbered paper cards, each with two fields: One field name, one field next: In the name field you can fill in a name of a friend, in the next field you fill in the number of the next card. The last card in the chain gets no entry in the next field.

Now we create a small Nim program which reads in names of our friends from the terminal, creates a list of all friends, and finally prints the list.

1 The actual name for this temporary variable is arbitrary, we could have used el for element maybe.

This example code seems to be not that easy. But it is not really difficult, and when you have understood it, you can call yourself a Nim programmer already. Maybe you should think about the code above for a few minutes before reading the explanations below.

First let us summarize what our program should do: It should read in some names of our friends which we would like to invite to our next party. Of course when entering the names, we would need a way to tell that we are done. In our program we can do that in two ways, we can enter an empty name by just pressing the return key, or we can enter the text "quit" to stop the loop. Unfortunately that means that we can never invite a friend with that name to our parties. When we have terminated the input loop, then the next loop prints all the entries to the terminal.

Let us start with the type and variable declarations: We use a user defined type named Friend which is a reference to an object , that object type has a field name of type string, and a field next which is again a reference to the same data type.

We are using two variables, one called n of type string to read in a name or the quit command from terminal, and a variable called f of type Friend. The variable f seems to match only to one single friend, but as the type of f has a next field it can be a whole list of friends, with f being the start or head of that list.

In the code above we are using a special while loop — special because the construct while true: and because the loop contains a break statement. Earlier we said that we should avoid the break statement in loops, because it interrupts the control flow and can make it more difficult to understand and proof the flow. But in this case that form makes some sense: For the first loop we have to first read in a name from terminal and then we can decide what to do, so we can not really evaluate a condition after the while statement at the top. So we use the simple constant condition true, which would never terminate the loop. We need a break inside the loop body to terminate the loop.

Let us investigate the second loop first as it is really easy: In the while condition we check if current value of f is nil, that is if there are no more entries in our list. For that case we terminate the loop, as we are done. If f has not the value nil, than f points to a valid content, that is there is at least a valid name, which we access by the field access operator and print it with echo f.name. Note that in Nim the field access operator . works in the same way for value objects types as well as for ref objects types. For ref objects types we could also write f[].name instead of plain f.name, that is we first apply [] to f to get the content, and then use the . operator to access the name field. In some other languages like C we would have to use a special operator -> to access fields of pointer or reference types.

The most interesting statement in the output loop is f = f.next. We assign the content of f.next to f and proceed with that new content. The content could be a valid reference to one more Friend object, or it could be nil, indicating that our loop should terminate.

The input loop is also not that complicated: To make the process of adding more friends to the list easy, we always add the new names at the beginning. First we ask the user to enter a name. We use write(stdout) for this, as echo() always generates a newline, but we want to read in the name on the same line. If the name is empty or has the special value "quit" then we terminate the input loop. In the loop we are using a temporary variable called node of type Friend, we allocate a memory block for it with new().[29] Then we assign the read in friend’s name n to the name field. The last two statements of the loop body are a bit demanding: First we assign to node.next the value of f. Now node is basically the start of our list, and its next field refers to the first element of the current list. Fine, but we said that the node variable is only a temporary variable, we do not intend to use it longer as necessary. But currently node is so useful, it is the head of our list. On the other hand, the former list start f is now useless, current f is identical with node.next. So the trick is, we just assign to f the value of node. Now f is the complete list, and we do not need node any more. The node variable can be used in the next loop iteration again, but we have to allocate a new memory block for the node reference, as the previous memory block is still in use, it contains the name which we just entered and also a reference to the next object in the list.

Note that we add the new elements at the top of the list in this way. We have done it that way because it is very easy in this way. For adding at the end of the list, we would have to use one more reference variable which allows us always access to the current end of the list, or we would have to traversal the list from head to tail whenever we would like to add elements at the tail.

For one more exercise let us consider deleting entries in our list. Basically that operation is very easy, we would just skip one entry. Lets adds this code to the program above:

Here we are using again an outer while loop to read in the names which we want to delete. That loop uses the condition while f != nil: because when the list is empty we should stop of course.

In the loop body we have an if statement, and in the else branch of the if statement we have one more loop. The reason why we need the if statement is, that the case that our name to delete is the first in the list is some sort of special. Let us investigate the inner loop first. That loop assumes that there are at least 2 elements in the list, f and f.next. We compare the name of the next entry with n. If they match then we would have to skip the next entry. We can do that by the statement f.next = f.next.next. That is we replace the reference from the current element f to the next list entry, that is f.next, by the next entry of the next element, which is (n.next).next. We do not have to write the parenthesis. The n.next.next entry can be nil, in that case it is the end of the list. If we found a matching name then we terminate the inner loop with a break statement, and we are done. Otherwise we assign to f the value of f.next and continue the loop execution. Now to the special case that the name to delete is the first in the list. We need the first if branch for that — if already the first element matches the name to delete than we just skip the first element by setting the head of the list to the next entry, which may or may not be nil.

This is one way to solve the task, for operations on lists there exist in most cases various solutions, some optimized to easy or short code, some for performance. You may copy the code segment above to the end of the former code, and maybe add one more copy of our printing loop at the end again. Then you should have a program that reads in a list, prints the contents, then ask for names to delete, and finally prints the resulting list. Maybe you can improve the code, or maybe you can detect special corner cases where it may fail. What is for example when some of your friends have the same name? May the program fail in that case? Or you may add more fields to your Friend data type. Maybe a textual field with content male or female, and you can report the ratio of male to female. And maybe remove males from the list when we have more males then females?

For references to objects the assignment operator = copies the references, but not the object. In the same way the operator == for equality test compares the references, but not the content of the objects to which the references point. If you want to compare the content of the objects, you can apply the dereference operator [] on both references:

We call or invoke a proc by just writing its name followed by a parameter list enclosed in parenthesis. The parameter list can be empty. When we call a proc, then the program execution continues with that procedure, and when the execution of the procedure terminates, then the next statement after that proc call is executed. Sometimes we say that we jump into a procedure and jump back when that procedure terminates.

In Nim functions are a special form of procedures that return a result and do not modify the current state of the program. Modifying a global variable or performing an input/output operation would be examples for modifying the state. We have already used some predefined procedures like echo() for output operations, add() for appending single characters to strings, and readLine() for reading in textual user input. And we talked about math functions like sin(), cos(), pow() — these are functions as they accept one or two arguments and return a result but do not change a state — calling them again with the same arguments would always give the same result. ReadLine() is only a proc, not a function, as the result may be different for each call, and as we pass a file variable as argument, which may change its state for each call, maybe because the file end is reached. A function is only a special sub-type of a procedure, the func keyword indicates to the reader of the code and to the compiler some special properties, that is that a result is returned and that global state is not changed. Whenever the func keyword is used a proc would do as well, and in this text we generally speak about procedures, even when a function would do.

Let us start with a very simple function called sqr() for square.

A procedure declaration consists of the keyword proc, a user selected name, a optional parameter list enclosed in parenthesis and an optional colon followed by the result data type. For a function declaration we use the keyword func instead of proc, and as functions does always returns a result, we have always to specify the result data type.

Note that this is only a declaration so far — the compiler could recognize the construct, its parameters and its result type. Sometimes we call this construct a procedure-header.

Generally we do not only declare a function, but we define it, that is we add an equal sign to the procedure header and add an indented procedure body that contains the code that is performed for each invocation.

Pure proc declarations can be necessary in rare situations, maybe when two procedures call each other. In this case the procedure defined first would call the other procedure, which is not already defined, so the compiler may complain about a unknown procedure. We could solve that problem by first declaring the second procedure, so that the compiler would know about it existence. We would then define that second procedure later, that is closer to the end of the program file.

The sqr() proc above accepts an integer argument and returns its square of same data type. We would call that proc like

Earlier in this book we said that the compiler processes our source code from top to bottom, and that the final program is executed from top to bottom too. The first statement is indeed true, for that reason it can be necessary to declare a function at the top, and define it below, as we can not call a proc before it is declared or defined.

For the program execution we have to know that procs are only executed when we call them. That is, when we write a proc at the top of our source code, then that proc is processed by the compiler, but it is not executed during program runtime before we call it. Actually, as the Nim compiler supports "death code removal", code of procedures that we never call would not make it in to our final executable at all.

Parameter lists of procedures consist of one or more lists of parameter names, separated with commas, followed by a colon and the data type of the parameters. The sub-lists with same data type are separated by semicolons:

While the wirthian languages would require semicolons to separate the parameter blocks, in Nim we could also use plain commas for that. For the data types of proc parameters all of Nim’s data types are allowed, including structured types, ref, pointer and container types, and we can pass literal values, named constants or variables.

When we call such a proc with multiple arguments, we have to specify the arguments in the order as they are listed in the proc header, separated with commas, and the arguments must have compatible data types:

Here compatible data types means, that for the i, j, and k parameters which are specified as int type in the proc definition, variables of smaller int types like int16 would work. For the two parameters of float type, we would have to pass floating point variables or a float literal. As a special case an int literal would work also, as the compiler knows the desired data type and automatically converts the int literal into a float for us, as long as that is possible without loss of precision. We could pass 2 instead of 2.0, but passing a very long int literal with more than 16 digits may fail at compile time:

Actually float32 types and int literals up to ten digits seems to work for float parameters, but even on 64 bit systems the int64 data type is not allowed for int parameters. As you see from the example above, it is possible to pass the same variable multiple times as a parameter, and empty string literals are of course allowed too.

Nim does also support default values for proc parameters and named parameters, that is that we can leave parameters unspecified and use the default value, or use the actual parameter names like in an variable assignment when we call a proc:

Here we used named parameters when calling the proc p(), this way we can freely order the parameters, and as parameter s has a default value, we can let it unspecified and just use the default value.

Functions always return a result, and procedures can return a result, but they don’t have to. In the C language function results can just be ignored, but in Nim whenever there is a result, then we have to use it at the call site, that is we have to assign the returned value to a variable or we have to use it in an expression. Nim enforces this, as generally the returned value is important, the returned value may be the actual result as in a sin() call, or it may give us additional information, like the number of read characters when we do text processing or maybe an error indication, like end of file. For the rare conditions when we really intend to ignore the result of a function call, we can call that function as discard myProcWithResult(a, b,…​). Another solution is to apply the {.discardable.} pragma to the function definition, we will learn more about pragmas later. When a procedure should not return a result, then we can use the void return type or just leave the return type out — the later is recommended, void types are used only rarely in Nim. When the proc has no parameters at all, then we can even leave out the empty parameter list in the proc definition:

Calling Procedures

var i = myFunc(7)
var j = myF()
var p = myF # not a function call, but assignment of the proc to variable p

proc p(i, j: int) = i + j
echo p(1, 2) # ordinary proc call
echo 1.p(2) # method call syntax
echo p 1, 2 # command invocation syntax
echo p (1, 2) # argument looks like a tuple, so this would not compile

Procedure Parameters of Var Type

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

# we can call that proc like
var message = "Hello World"
frame(message)
echo message

func framed(s: string; c: char = '*'): string =
  var res = newStringOfCap(s.len + 4)
  add(res, c)
  add(res, ' ')
  add(res, s)
  add(res, ' ')
  add(res, c)
  return res

# we can call that proc like
echo framed("Hello World")
echo framed("Hello World", '#')

Click to see a possible solution

from sugar import dup

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

echo "Hello World".dup(frame)
echo "Hello World".dup(frame, frame)
echo "Hello World".dup(frame('#'))

* Hello World *
* * Hello World * *
# Hello World #

Returning from a Procedure and the implicit Result variable

func sqr1(i: int): int =
  i * i

func sqr2(i: int): int =
  result = i * i

func sqr3(i: int): int =
  return i * i

func sqr(i: int): int =
  result = i
  i * i

Var Return Type

var g = 0
proc writeAccessToG(): var int =
  result = g
writeAccessToG() = 6
assert g == 6

Proc name overloading

func sqr(i: real): real =
  i * i

Objects and Ref Objects as Procedure Parameters

type O = object
  i: int

proc t1(o: O) =
  o.i = 7 # Error: 'o.i' cannot be assigned to

proc t2(o: var O) =
  o.i = 13

proc main =
  var x = O(i: 3)
  echo x.repr
  t2(x)
  echo x.repr

main()

[i = 3]
[i = 13]

proc t1(s: string) =
  s.setLen(7)
  s[0] = 'x'

proc t2(s: seq[int]) =
  s.setLen(7)
  s[0] = 13

type O = ref object
  i: int

proc t1(o: O) =
  o.i = 7

proc t2(o: var O) =
  o = O(i : 11)

proc main =
  var x = O(i : 3)
  echo x.repr
  t1(x)
  echo x.repr
  t2(x)
  echo x.repr

main()

ref 0x7f054a904050 --> [i = 3]

ref 0x7f054a904050 --> [i = 7]

ref 0x7f054a904070 --> [i = 11]

The openArray and varargs data types can be used only in parameter lists. OpenArray is a type which allows to pass arrays and sequences to the procedure or function. Allowing that makes sense as arrays as well as sequences store their content in a block of memory, which can be processed uniformly. Although arrays generally do not have to start with index number 0, when passed as openArray the first element is mapped to index 0, and the index of the last element is available by using the high() function on the passed array parameter. Whenever we write a procedure that accepts an array or a sequence, we should consider using the openArray parameter type to allow passing in both data types. Strings can be passed also to procs accepting openArrays with char base type. Note that a proc with openArray parameter type can not change the length of a passed seq, as for the openArray parameter type sequences are handled like arrays. So in the code below proc t1 generates a compiler error while t2 compiles and works fine.

The varargs parameter type is similar to the openArray type, but it additional allows passing an arbitrary number of single arguments. The compiler automatically collects the single arguments into an array for us, so in the proc body we can use it like an array, e.g. iterating over it.

There exists a variant of the varargs argument type that performs a type conversion automatically by applying a proc on all arguments. For example varargs[string, $] would apply the stringify operation on the passed arguments automatically. That is what echo() does.

Varargs arguments may be only allowed for the last argument in a parameter list.

Finally we may wonder if it makes sense to specify a parameter of type var varargs. If we try to pass a constant string this will obviously not work, and if the compiler generates an array for us it does also not work, the automatically generated array seems to behave like a constant array. But may we pass an array variable? Let us try:

Surprisingly that does not compile, while it works when we replace varargs with openArray.

In some other programming languages like Python or Ruby we can define class methods or static methods which are bound to a class or type and can be called as MyType.myProc. In Nim we can do something similar by use of the typedesc proc parameter type:

Here we used the method call syntax instead of start(Factory). We will learn more about the typedesc data type later.

Scoping, visibility and locality is an important concept in computer programming to keep the source code clean. Imagine that a variable which we declare at some point in our program would be visible everywhere. That would even for medium size programs generate a lot of confusion — whenever we would need a variable we would have to carefully check which names are already in use. And for performance it would be bad also, as all variables declared somewhere would reside permanently in memory.

So most programming languages including Nim support the concept of locality — names declared inside of a procedure body or inside another form of block are only visible there and can only be used there. We say that they are only visible in that scope. For Nim we can say that whenever Nim’s syntax requires a new level of indentation, that is a new statement block, then all symbols declared in that block are only visible in that block and in sub-blocks of this block, but not outside of that block. Nim has another important concept of visibility, which is called modules and allows separation of our code in logically separated text files with well defined visibility rules, we will discuss modules later.

Visibility is really a simple concept, let us regard this useless example:

In line one we declare a so called global variable, that one is visible after declaration, that is below the line where it is declared, in the whole program. The variables declared in the proc p1 are called local variables, they are not visible outside of that proc p1. The variable x is declared at the start of the proc body and is visible in the whole proc everywhere, while variable y is declared in the if block and is visible only there. So it should be clear if the last two echo statements for x and y compile fine? Remember that symbols that we define inside of a new scope may shadow symbols that were visible outside of the actual block, e.g. by defining a variable named e of arbitrary type in the proc p1 from above would shadow the global variable e, that is the global variable e would become invisible until execution of proc p1 terminates. We discussed shadowing already in the introducing section Scopes, Visibility, Locality and Shadowing.

Related to visibility of variables is their lifetime, that is the duration how long they exist and how long they can store a value. Global variables exist for the whole program runtime — when you have assigned a value to it that value can be used everywhere as long as the program runs, and as long as you do not assign a different value of course. Global variables are generally stored in a special memory region that is called the BSS region.

Variables of value type defined locally inside a procedure or function do exist only for the execution time of that proc, that is they are created when the proc is invoked and vanish when the proc terminates, that is when execution continues with the statement following on the proc call.

Local variables declared in a proc reside in a special memory region of the RAM which is called the stack. The stack is nothing more than an arbitrary part of the hole RAM that is used in some clever fashion: The memory words in it are used in consecutive order. A so called stack pointer is used to indicate the address of the first free area in that stack. So when a proc is called, which may have n bytes of local variables, then the compiler can use the area where the stack pointer points to for that variables, and when the proc is called then the stack pointer is increased by that size. So the stack pointer points again to the next free area of the stack, and another proc can be called in the same way from within the current proc. Whenever a proc terminates, the stack pointer is set back to the value which it had when the proc starts execution. This method of memory management is simple and fast, but it does only work when the total amount of memory that the local variables in a proc needs is known at compile time, so that the compiler can adjust the stack pointer accordingly. It does not work for dynamically sized data types like strings or sequences.

Note that pointers and references are value types itself, we can regard pointers and references as a plain integer variable interpreted in a special way — as a memory location. But the memory blocks to which the pointers and references may point and that is allocated by alloc() or new() is different: That memory blocks are not allocated on the stack, but in the ordinary RAM which we call heap to separate it from the stack.

So why can the stack not be used for memory blocks which alloc() or new() provides for us: An important fact for the use of the stack to store variables is that the total size which is needed by a proc for all the static variables must be a compile time constant. The stack pointer is adjusted by that amount when the proc starts and all the local variables are accessed with a fixed offset to that stack pointer then. When we use alloc() or new() in a proc, then we may call that multiple times like we did in our previous list example, and for alloc() an additional fact is that the byte size that alloc() should reserve can be a runtime value. So the total amount of RAM that alloc() or new() would allocate is a runtime value, and we can not use the stack for it. Instead alloc() and new() allocates block of memory in a more dynamic fashion, which is basically that they ask the OS for a free block of right size somewhere in the available RAM. That block is later given back to the OS for reuse by functions like dealloc() or automatically by the GC.

Let us at the end of this section investigate some special cases:

While in languages like C we have always a well defined main() function and all program code is contained in this function or in other functions which are called from this main function, in Nim we have also global code as in scripting languages Ruby or Python:

It should be clear that the global variable i resides in the BSS segment. But what is with the variable j declared in the body of the while statement? It is clear that that variable is only visible inside of the body of the while statement. But does j reside on the stack? There seems to be no proc involved so there may be no stack? The variable j may reside in the BSS segment too? That is not really clear and may be different for different Nim compilers maybe. But why should we care for that detail at all? Well it may be important for performance. Local proc variables allocated on the stack are generally optimal for performance, and they are optimized by the compiler very well. We will learn more about the reasons for that later when we discuss the data cache. For now we should only remember that it may be a good idea to avoid global code and put all code in procs. We may have an arbitrary named main() proc then and call that from global scope only. At least for the current Nim v1.6 that seems to be a good idea, maybe later versions or other implementations will automatically move all global code into a hidden proc for us.

For optimal performance put all your code in procedures or functions and avoid global code and when possible global variables.

Let us discuss above while loop again, but this time in the body of a proc:

When we carefully investigate that proc within the while loop we may wonder about two points. First we said earlier that we can and should use the let keyword instead of var when there is only one assignment to a variable, so the variable can be regarded as immutable. But the loop is executed 100 times, so how can we say there is only a single assignment to variable j? The trick is, that j is locally to the while loop, and that j is virtually newly created and initialized to 0 for each iteration. So let is OK and the compiler does not complain.

We can test that fact with this simple program:

The output is 1 for each loop iteration, as variable a is virtually newly created for each loop iteration.

We said virtually newly created, because we can not be sure how the compiler may handle it internally. Is storage for variable a already allocated when the proc is invoked, that is in the same way as storage for the loop counter variable i is allocated on the stack when the proc is called. Or is storage for variable a reserved for each loop iteration by increasing the stack pointer at the start of the loop and resetting it at the end of the loop. We can not be sure without reading the compiler source code, but finally we should not care, as it does not really matter.

In the previous section we defined a sqr() proc for ints and one for float numbers. Both procs look nearly identical, only the data types differ. For that case we can use so called generic procedures.

We put a square bracket after the function name which includes a symbolic name, and that name is then used instead of concrete types in the proc header or in the proc body.

We can now call that proc with parameters of different types including int and float types. You may wonder why that works — Nim is a statically typed language, so how can the parameter of function sqr() as well accept an integer as a floating point number? Is there a hidden type conversion involved? No, the trick is that whenever we call that generic proc with a different type, then a new proc or function is instantiated. As we called the generic sqr() proc with an int and a float parameter, during compile time the compiler creates machine code for two separate functions, one which is called when an int is passed as parameter, and one which is called when a float is passed. If we would call that proc name again with an int or float parameter, than one of the two existing procs would be used. But for a different, still unused data type like float32 again a new proc would be instantiated. In this way generics procs can lead to some code bloat. Note that calling the generic function with a data type like a character or a string would fail, as that types do not support multiplication with itself.

A slightly different notation is available by so called or types:

Here we have limited the parameter types to the int type or the float type. We could have defined also a custom type first, like type MyNum = int or float and use that type for the type of our sqr() proc. These or types are also called type classes. Instead of keyword or the | character can be used for defining type classes. Again the compiler would instantiate two separate functions for the both data types. As we had not the symbolic type T available here, we have used the keyword auto as return type, and for the type of variable p we used the macro typeof(). The type auto for the return type works as long as the function returns a well defined type. Note that we can not decide at runtime what a type the function should return, so a construct like if cond: return 2 else: return 3.1415 would not work, at least not when the values are variables of different type. For the literal value it may work, as the compiler may be smart and guess that we want to return the float literal 2.0.

A bit care is needed when we define procs for mutable or types:

Here we try to define a proc called t which should accept a mutable seq[uint8] or a mutable seq[char] as parameter. While the first line compiles fine, the seq[char] would be immutable. The correct notation is shown in the second line. This behavior was labeled "won’t fix" in github issue tracker, so we have to remember this case, see https://github.com/nim-lang/Nim/issues/15063#issue-665553657.

Let us assume that you want to define a proc that accepts two numbers of type int or float and that returns a float. You may write it in one of this ways:

The commented out lines would give you a compiler error. The reason for this is that the proc sqrsum2[T] defines a generic proc, but the compiler enforces that both parameters have the same type. The expression sqrsum2(x, 2) compiles fine, as due to the first parameter x the compiler instantiates a proc for a float parameter type, and then converts the second parameter, which is an integer literal, to float automatically. This automatic conversion is only done for literal numbers, not for variables. The expression sqrsum2(2, x) does not compile, as due to the first parameter, which is an integer literal, a proc for integer parameters is instantiated, and the second x parameter of [.type]float type is not compatible with the instantiated proc.

Generics can become a bit complicated, as we may use multiple different generic types for different proc parameters. And we can use generics also for object types, we may for example create lists like we did for our names list that work not only for strings, but that can work with other data types like numbers or sequences in a very similar way. We may explain that in more detail later.

Generics are used a lot in Nim’s standard library. Most container types like sequences or tables accept generic types, and generic procedures like sort() are provided which can easily sort arbitrary data types and objects. We have only to provide a cmp() proc for our user defined data types which sort() can call to compare the values during the sorting process.

We will demonstrate the use of generics for library modules with a few tiny examples: Assume we create a library which should be able to store and process arbitrary data types. The stored values may have well defined relations, which enables ordering or much more complicated spatial relations. Triangulation of spatial data points or grouping the data in structures like RTrees for fast point location as well as geometric processing with algorithm like finding the convex hull are some examples. To make our example simple and compact, we define a generic container type which can store only two values of arbitrary data type. The container allows to sort the elements by size. The following code example defines a generic container called MyGenericContainer, a proc to add() data objects into the container instance and a sortBySize() proc to sort the two elements:

The sortBySize() proc of the above examples accesses the size field of our data objects directly, so we can use the container for arbitrary data types as long as the data types have a size field and as long as a > proc is defined for the data type of the size field. In the above example we have defined a $ procedure to convert instances of our container to a string, which allows us to call the echo() function on it. The output of our program looks like

We can avoid the restriction of a matching field name when we provide getter and setter procedures which the library procs can use to access the important fields:

In the example above our TestObj1 data type has no field with a name matching for the sortBySize() proc, but we define a size() proc for our data type which that library function can use. This solution is more flexible, and when we add the inline pragma to the used size() proc or when we compile with link time optimization (LTO) enabled, then the overhead should be negligible. The two examples above are located in a single file each, but of course for practical use we would use separate modules for the library and the application part as in

The example with direct field access would look for different modules like this:

You may wonder why we do not have to export the size field of our TestObj1 (or maybe the object itself also) as it is used from code defined in a different module. The reason why we do not need export markers is that the sortBySize() is defined in the library module, but as it is a generic procedure, it is instantiated and executed in the application module. For the same reason we had not to export the size() getter procedure before.

Finally one more way to use generic library modules is by passing procedure variables to the library functions. The passed in procedures may provide access to properties or attributes of the stored objects, or they may offer relations between the objects. The later is often used for sorting purposes:

Here we have modified the sort() proc of our library module in a way that it takes an additional procedure parameter. In this case we use a procedure signature that takes two object instances and returns a boolean value indicating if the first parameter is smaller than the second. In our application module we define a matching procedure and pass that one to the sortBy() procedure. Again we get the desired sorted output:

This last method is used often in Nim’s standard library, e.g. for sorting sequences with custom objects. Unfortunately this way can introduce some performance regression, as the procedure variable has to be passed to the called proc and so inlining of that passed proc is not possible for the compiler. [32]

A useful coding style introduced by OPP languages is the method call syntax, which was initially used in OOP programming style for objects, and later applied by languages like Ruby to all data types. Ruby in some way regards all data as objects.

Method call syntax means, that for example for a variable s of data type string we do write s.add(c) instead of add(s, c). Or for an integer variable i we may write i.abs instead of abs(i). That is we put the first parameter of the proc parameter list in front of the proc name, and separate that parameter from the proc name by a period. The compiler regards both notation as equivalent. The advantage of the method call syntax is that we may save a character and that it is more clear with what "object" we are working, as it stands in front of the expression.

Most OOP languages allows that notation only for a class, for example the string class may declare all possible operations that can be done with strings, and the method call syntax is used for that operations. One problem is, that it can be difficult to add more operations which can be used in that style, as often all that operations are defined in the class scope. Ruby fixed that restriction by allowing so called reopen of classes, that is user can later add more operations.

Nim simple allows that notation generally, as did the D language, but D used the term Uniform Function Call Syntax (UFCS) for it.

Procedures and functions are not always fully static entities. We can assign procedures and functions to variables, and we can pass whole procedures or functions as parameters to other procedures or functions. And functions can even generate and return new functions. Let us investigate how procedure variables work:

The output of the two echo statements should be 14 and 49 — we called in both cases the same proc variable with the same parameter, but the proc variable p was in the first call an alias for p1, and in the second call an alias for p2. Note, when we assign a proc to a proc variable we do only write the name of that proc, there is no () involved. That is because we assign that proc to the proc variable, but we do not call the proc in this case. Of course, when we assign a proc to a proc variable then the proc signatures have to match, that is the parameter list and the result have to be compatible.

Now we use a function as a proc argument.

A common use case for a function as a proc parameter is sorting. We can use the same sort procedure for different data types when we provide a cmp() proc that can compare that data type.

The sort() procedure is provided by the algorithm module. The sort() proc accepts an array or a sequence, and a cmp() proc that gets two parameters of the same type as the elements in the passed array, and that returns -1, 0, or 1 as the result of the comparison. We could easily sort other data types like strings or our custom objects by an arbitrary key, as long as we can provide a matching cmp() proc. For the cmp() proc it is important that it returns a well defined result based on the input, and when both parameters are equal it should really return 0. If you would exchange the return values 1 and -1 in the cmp() proc above, you would invert the sort order.

While in C all functions must be defined in top level scope and nesting of functions is not allowed, in Nim procedures can contain other procedures. A special case occurs when the sub-procedures do access variables of the outer scope. In this case the sub-procedure is called a closure:

When you run this program the output should be

This program is not that easy, but when you think about it a bit you should be able to understand it. The task is to extract from a string all the digits and to ignore the other characters.

To get the digits, we use a local procedure that uses the pos variable of the enclosing procedure, and also access the parameter s of the enclosing procedure. The closure nextDigit() checks if the position in the string is still valid, that is if it is still smaller than the length of the string, and also checks if the current character is a digit. The first check uses the standard procedure len() which return the length of a passed string parameter, that is how many characters the string contains. We have used the method call syntax here instead of using the ordinary proc call len(s). The next check test if the current character is not a decimal digit. For that test we could use a series of compares like if c == '0' or c == '1' or …​ or c == '9'. But to make such tests easier and faster, Nim offers one more data type, the set type. And the notin operator tests if an value is not contained in a set constant. An important point for the expression after the while statement is, that it is processed from left to right. That fact is here important, because we have first to check if pos is still a valid position, before we can use the subscript operator [] to access the current character and test if it is not contained in the set. If the check for the valid position would not come first, then we may access an invalid position in the string and we would get a runtime range error.

While the position is still valid but the current character is not a digit we increase the position. The while loop can end by two conditions: Either the current character is a digit, or we have reached the end of the string and we have to stop. For the last case we use a special stop mark, we return a special character which we have entered in escape notation as '\x0'. That is a very special character, that is used in C to mark the end of strings. It is the first character in the ASCII table and has the decimal value 0. We said earlier that characters are encoded in 8 bit and correspond to the unsigned integer numbers 0 up to 255. '\x0' is just a special notation for the first character which corresponds to integer value 0. Well, when the end of the string is reached then we return that character. Otherwise we return the current character. Remember, from the while condition we know that the string end is reached, or current character is a digit. As we tested for the string end before, we can only have the case that the current character is a digit now. But can we return that character immediately now? If we would, s[pos] would be a digit, and we would get exactly the same character for the next proc call! So we have to move to the next character by increasing pos before we return that character. For this the pre-declared result variable is useful. We assign the current character to the result variable, and then increase pos. As the last statement in our proc is not a expression but a plain inc() statement, the content of the result variable is returned. The other while loop in the outer procedure is very simple, we just call the closure in the body of the while loop and terminate the loop when we get the special Null character.

And finally an example where one proc returns another procedure:

The output of echo() would be 9 in this case. This construct is sometimes named currying.

In the section Module sequtils in part III of the book we will introduce a few functions which are often used in the functional programming style like map() or filter(). These functions get procedures as arguments that determine how container data types are converted. We can pass a regular named procedure as second argument to procs like map() and filter, or in simple cases we can just pass an anonymous proc or use the ⇒ operator provided by the sugar module:

Here we use the toSeq() template to create our initial sequence with numbers 0 up to 9 — just to have not to type all the number in, we will explain templates soon. Then we apply the filter() proc to that sequence. The filter() proc expects as second argument a function with an argument of the seq’s base type returning a boolean value. We can pass the named function primeFilter(), or we can just pass an anonymous proc explicitly.

In the last two lines of our example, we use the map() function to convert the data of our sequence. Map() expects as second argument a proc with a parameter of the seq’s base type returning a result of the same type. In the line before the last one, we specify an anonymous proc as parameter, while in the last line we use the ⇒ operator from the sugar module to just specify the actual conversion.

When a function is called only with constant arguments, then the compiler can execute it already at compile time:

Here we used a function called genSep() to create a string constant at compile time. When we compile above program, we get the message "Generating separator string". As that proc is not executed at program runtime, it is not included in the final executable program. Here we had to use the debugEcho() proc instead of the ordinary echo(), because echo() is not really a pure function, and the compiler would complain when we use echo() in a pure function. DebugEcho() is not really pure as well, but the compiler ignores that fact, which is OK for debugging purposes. We could even make gesSep() a plain proc and then use echo(), the compiler would not complain. But it would complain, if for instance we would access global variables from inside the genSep() proc.

Calling procedures and functions is always some overhead — proc parameters may have to put on the stack or loaded into CPU registers, some CPU or FPU registers may have to be saved, the stack pointer and the program counter have to be updated and finally the instruction cache has to be filled with new instructions.

So for tiny procs actually calling the proc may take more time than processing the actual code inside of the proc. To avoid this additional effort, procedures and functionss can be inlined. The compiler may do this automatically for us, but we can support it by applying the {.inline.} prama to tiny procs.[33] For inlined procs the code is just inserted directly at the callsite. This may increase the total executable size, when the proc is used often. So we should use the inline pragma with some care. Another option is to just compile the whole program with link time optimization passing the option -d:lto to the compiler, that way the C backend can automatically inline all proc code, even procs from imported modules. One more option is to use templates instead of tiny procs — templates do always a plain code substitution, so templates may behave very similar to inline procs. We will discuss templates later. The following example show how we can apply the inline pragma to procedures and functionss:

Note that functions from shared libraries can not be in lined, so calling external C functions, directly or indirectly, may be slower than expected.

Procedures and functions can call them self in a repetive manner, which is called recursion. Obviously there must exist some condition that finally stops the recursion, without the proc would call itself again and again, for each call some data would have to be stored on the stack, at least the proc return address, so finally the stack would overflow and the program would crash. Generally recursion should be used only for cases where it really helps to simplify the algorithm. In part V of the book, in the section about various sorting algorithm, we will discover some useful applications for recursion. In most cases a plain iterative algorithm is faster than a recursive one, because all the overhead with many proc calls is avoided for plain iterative solutions. But sometimes recursive algorithm are easier to understand, or programming an iterative solution may be really complicated.

As one of the most simple algorithm we will present here the recursive fac() function:

That function should terminate, as we only call itself again with a decreased argument. Of course, using recursion here is not really smart, it should be easy for you to convert the proc into an iterative solution without recursion. Note that recursive procs can never be inlined!

Nim’s converters are a special variant of functions that are called automatically by the compiler when argument types do not match.

With above converter we can pass an integer to a proc that expects a boolean parameter, and we can even use an integer as a logical expression in an if condition in the same way as it is done in C language. Converters do work only in a direct way, that is automatic chaining is not supported: If we have one converter from character to integer and one from int to boolean, that does not mean that we can pass a character to a proc that expects a boolean. We would have to declare one more converter that directly converts character to boolean.

Whenever we do consider using converters we should think twice — converters may be confusing, may have some strange effects and may increase compile time.

Maybe you wondered why we wrote above converter in such a verbose way? Well it was done intentionally, but you are right of course, we can write it just as

type
  Shape = ref object of RootRef

  Rectangle = ref object of Shape
    x, y, width, height: float

  Circle = ref object of Shape
    x, y, radius: float

  LineSegment = ref object of Shape
    x1, y1, x2, y2: float

method draw(s: Shape) {.base.} =
  # override this base method
  quit "to override!"

method draw(r: Rectangle) =
  echo "drawing a rectangle"

method draw(r: Circle) =
  echo "drawing a circle"

method draw(r: LineSegment) =
  echo "drawing a line segment"

proc main =
  var l: seq[Shape]
  l.add(Rectangle(x: 0, y: 0, width: 100, height: 50))
  l.add(Circle(x: 60, y: 20, radius: 50))
  l.add(LineSegment(x1: 20, y1: 20, x2: 50, y2: 50))

  for el in l:
    draw(el)

main()

drawing a rectangle
drawing a circle
drawing a line segment

Tuples are heterogeneous container types similar to the struct type in C. As Nim’s object type creates no overhead as long as we use no inheritance and so also directly corresponds to the C struct type, tuples are very similar to Nim’s objects. The biggest advantage of tuples is, that we can create anonymous tuples, and that Nim supports the automatic unpacking of tuple variables into ordinary unstructured variables.

Compared to objects, tuples do support no inheritance at all, all the tuple fields are always visible, and different tuple types are regarded as identical, when all the field names and field data types match. Remember that two different object types are always distinct in Nim, even when the actual type definition looks identical.

We can define tuple types in the same way as we define objects, or we can use the tuple[] constructor. Additional we can define anonymous tuples just by enclosing its field types in round brackets. The fields of tuple types can be accessed by field names as we do it for objects, or we can access the fields with constant indices starting at zero.

In the code example above we show two equivalent ways to define a tuple type, but actually we do not use that type at all but return an anonymous tuple from our proc, that is a pair of an int and a bool.

Using automatic tuple unpacking and type inference our dst and check variables gets the data types int and bool.

Tuples are also useful when a function has to return a value and also an error state, or if it may not be able to return something at all in a special case. For reference types we could return nil then, but for results of value type like int or float we may not have a well defined error indicating constant, so we can return a tuple with an additional bool indicating success or error. But of course we could use exceptions instead, or we could use Nim’s option type instead. We will learn more about that later.

Here are two examples which uses a tuple as a proc parameter:

Proc p1() creates a tuple type using the tuple constructor syntax with named fields, so in the proc body we can access the fields by its names, while proc p2() uses an anonymous tuple, and so has to access the fields by constant indices. Both procs are called with an anonymous tuple parameter.

Nim’s object variants, sometimes also called sum types or abstract data types (ADT), are an advanced and type save variant of the union type known from C. The basic idea is that we may use value types that may store similar, but not identical data. Untyped languages like Ruby or Python allow that of course, and we can do it in Nim with ref types and inheritance too, as we showed in a previous section with our Shape base type and various geometric shapes. We could store that ref types in arrays or sequences or linked list and use dynamic dispatch for processing the various sub-types. That is convenient but gives not maximum performance due to dynamic dispatch at runtime and due to bad cache use. So we may like to have a value type with different content, so that we can store all the value types in a seq and all entities reside in a compact block of memory for good cache use.

Objects variants can have common fields like the boolean state visible above, but the other fields are not allowed to use the same names, so we had to use x0 and y0 for the names of the center coordinates in the circle variant.

As you can see we can store all the different object variants as value objects in a sequence and iterate over it. Note that object variants may waste some storage, as all variants are silently enlarged to have the exact same size, so that all variant types can be stored in an array or sequences and can be passed as proc parameters in the same way to the same proc.

iterator decDigits(s: string): char =
  var pos = 0
  while pos < s.len:
    if s[pos] in {'0' .. '9'}:
      yield(s[pos])
    inc(pos)

for d in decDigits("df4j6dr78sd31tz"):
  stdout.write(d)
stdout.write('\n')

iterator items(a: string): char =
  var i = 0
  while i < len(a):
    yield a[i]
    inc(i)

iterator pairs(a: string): tuple[key: int, val: char] =
  var i = 0
  while i < len(a):
    yield (i, a[i])
    inc(i)

var s = "Nim is nice."
for c in items(s):
  stdout.write(c, '*')
echo ""
for i, c in pairs(s):
  echo i, ": ", c

iterator mitems(a: var string): var char =
  var i = 0
  while i < len(a):
    yield a[i]
    inc(i)

iterator mpairs(a: var string): tuple[key: int, val: var char] =
  var i = 0
  while i < len(a):
    yield (i, a[i])
    inc(i)

from strutils import toLowerAscii
var s = "NIM"
for i, c in mpairs(s):
  if i > 0:
    c = toLowerAscii(c)
echo s # Nim

var i = 0
while i < len(a):
  var c = a[i]
  echo c, '*'
  inc(i)

iterator myCounter(a, b: int): int {.closure.} =
  var i = a
  while i < b:
    yield i
    inc(i)

for x in myCounter(3, 5): # ordinary use of the operator
  echo x

echo "---"
var counter = myCounter # use of an iterator instance
while true:
  echo counter(5, 7)
  if counter.finished:
    break

3
4
---
5
6
0

var counter2 = myCounter
while true:
  let v = counter2(5, 7)
  if counter2.finished:
    break # v is invalid
  echo v

proc mycount(a, b: int): iterator (): int =
  result = iterator (): int =
    var i = a
    while i < b:
      yield i
      inc(i)

var c1 = mycount(5, 7)
for i in c1():
  echo i

echo "---"

var c2 = mycount(2, 5)
while true:
  let v = c2()
  if c2.finished:
    break # v is invalid
  echo v

5
6
---
2
3
4

#define PI 3.1416
#define SQR(x) (x)*(x)

const PI = 3.1416
proc sqr1[T](x: T): T = x * x
template sqr2(x: typed): typed = x * x

type
  Point = object
    x, y: int

  Circle = object
    center: Point

template x(c: Circle): int = c.center.x

template `x=`(c: var Circle; v: int) = c.center.x = v

var a, b: Circle

a.center.x = 7
echo a.center.x

b.x = 7
echo b.x

Error: in expression 'b.center.7': identifier expected, but found '7'

template `!=` (a, b: untyped): untyped =
  not (a == b)

const
  debug = true

template log(msg: string) =
  if debug: stdout.writeLine(msg)

var
  x = 4
log("x has the value: " & $x)

  if debug: stdout.writeLine("x has the value: " & $x)

template log(msg: string) =
  for i in 0 .. 2:
    stdout.writeLine(msg)

var x = 4
log("x has the value: " & $x)

template gen =
  var a: int
  proc maxx(a, b: int): int =
    if a > b: a else: b

gen()
echo maxx(2, 3)
# echo a

template withFile(f: untyped; filename: string; actions: untyped) =
  var f: File
  if open(f, filename, fmWrite):
      actions
      close(f)

withFile(myTextFile, "thisIsReallyNotAnExistingFileWithImportantContent.txt"):
  myTextFile.writeLine("line 1")
  myTextFile.writeLine("line 2")

from std/math import sqrt

template liftScalarProc(fname) =
  proc fname[T](x: openarray[T]): auto =
    var temp: T
    type outType = typeof(fname(temp))
    result = newSeq[outType](x.len)
    for i in 0 .. x.high:
      result[i] = fname(x[i])

liftScalarProc(sqrt)   # make sqrt() work for sequences
echo sqrt(@[4.0, 16.0, 25.0, 36.0])   # => @[2.0, 4.0, 5.0, 6.0]

Parameters passed to templates can have all the data types that we can use for procs including special types like openarray and varargs, and we can use as types additional the symbols untyped, typed or typedesc.

The typedesc type can be used to pass type information to the template, e.g. when we want to create a variable of a special data type. The "meta-types" typed and untyped are used, when we want to create some form of generic template, that can accept different data types. Actually the distinction between typed and untyped parameters is not that difficult and important for templates as it is for macros, in most cases it is just clear if we need the typed or untyped parameter type for a template, or both work fine. We discuss the differences between typed and untyped in much more detail in part VI of the book, when we discuss macros and meta-programming.

The following example demonstrates the use of the untyped and the typedesc parameter:

As the parameter n is untyped, the compiler allows us to pass an undefined symbol to the template. If we would change the parameter type to typed the compiler would complain with a message like Error: undeclared identifier: 'i'

For the second template called declareVar() we use an additional parameter of typedesc type, so that the template can create for us a variable of just the passed data type.

In the withFile() example above we showed that we can pass a block of statements as the last argument to a template following the special : syntax. To demonstrate the difference between code blocks of typed and untyped data type we will cite the compiler manual, see https://nim-lang.org/docs/manual.html#templates-passing-a-code-block-to-a-template:

Usually, to pass a block of code to a template, the parameter that accepts the block needs to be of type untyped. Because symbol lookups are then delayed until template instantiation time:

The above code fails with the error message that p is not declared. The reason for this is that the p() body is type-checked before getting passed to the body parameter and type checking in Nim implies symbol lookups. The same code works with untyped as the passed body is not required to be type-checked:

One more use case for templates with untyped parameters is the generation of math operations for custom data types. Let us assume that we have created a custom Vector object, for which we have to define addition and subtractions operations. Instead of writing code for both cases, we can use a template and pass the actual math operator as untyped parameter:

This works, because for operators math like 1+2 can be written as +(1, 2) and because we can pass such an operator as untyped parameter to a template.

For the advanced template stuff you should consult the compiler manual.

This includes the symbol binding rules, identifier construction in templates, lookup rules for template parameters, hygiene in templates, use of the inject pragma, and limitations of the method call syntax.

All this is explained well in the compiler manual, so it makes no sense to repeat it here. And it makes more sense to read it when you actually have problems with the (default) behaviour of templates in special situations.

var i: uint8 = cast[unint8](myBoolVar)

from strutils import toBin
var i = 1.int8 # 0b00000001
i = i shl 7 # 0b10000000
i = i shr 2 # 0b11100000 as sign is preserved
echo i.toBin(8)
var j: uint8 = 0b11111111
j = j shr 2 # 0b00111111, div 4 for unsigned int
echo j.int8.toBin(8)

var a = 3 # two rightmost bits, at position 0 and 1 are set
var b = a and 2 # extract bit 1, so b gets the value 2
b = a and 4 # extract bit 3, which is unset, so result is 0
b = a or (4 + 8) # result is \b00001111, decimal 15

proc charToInt(c: char): int =
  if c in {'0' .. '9'}:
    return ord(c) - ord('0')
  raise newException(OSError, "parse error")

proc main =
  while true:
    stdout.write("Please enter a single decimal digit: ")
    let s = stdin.readline
    try:
      echo "Fine, the number is: ", charToInt(s[0])
    except:
      if s.len == 0:
        break
      echo "Try again"

main()

type
  O = object
    i: int

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

type
  O = ref object of RootRef
    i: int

proc `=destroy`(o: var typeof(O()[])) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O = O() # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

type
  O = object
    i: int
  OP = ptr O

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = create(O) # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

type
  O = object
    i: int
  OP1 = ptr O
  OP = distinct ptr O

proc `=destroy`(o: var OP) =
  echo "destroying OP"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = OP(create(O)) # new O
      OP1(o).i = rand(100)
      echo OP1(o).i * OP1(o).i

randomize()
test()

81
destroying OP
3600
destroying OP
2401
destroying OP
9025
destroying OP

When we use OOP style programming with subclassing of ref objects, then it is useful to know that for subclassed ref objects the destructor of the parent class is automatically invoked when we do not define our own one for our subclassed type. This works also when we import the parent type from another module, at least since Nim v1.6:

When we compile module t.nim with --gc:arc or --gc:orc and run it, we get this output:

So when our variables o1 to o5 go out of scope, then the destructors are called. Module tt.nim defines a ref object type, but the destructor proc takes a var value parameter. The destructor is called when a value object or a ref object goes out of scope. Our variable o1 has type tt.O1, so it was indeed expected that its destructor from module tt.nim is called. Variable o2 is a ref object with parent O1, as we define no destructor for this type, the destructor of the parent type is called. The variables o3 and o4 are of ref object and of value object types, each with a field of type O1, and for that field the destructor for O1 is called. Finally for type O5 we define our own destructor, which then additional calls the destructor of module tt.

Destructors are mostly used for library implementations, e.g. for a file data type which is automatically closed when a file variable goes out of scope. As you may never have to use destructors yourself, it is not necessary to remember all these details. But it is good to know that destructors behave in a way as we may have expected, and when you later wants to use a destructor in your own code, you can consult again this section or maybe better the Nim manual.

References:

type
  O = ref object of RootRef
    i: int

proc finO(o: O) =
  echo "finalize O"

proc newO: O =
  new(result, finO)

proc main =
  var o = newO()
  var o2 = new(O)
  var o3 = O(i: 7)

main()

finalize O
finalize O
finalize O

# module tt.nim
type
  O* = ref object of RootRef
    i: int

proc fin*[T](o: T) =
  echo "finalize T"

proc newO*: O =
  new(result, fin)

import tt

type
  OO = ref object of tt.O
    x: float

proc finn[T](o: T) =
  echo "finalize O"

proc main =
  var oo: OO
  new(oo, finn)

main()

Error: type bound operation `fin` can be defined only in the same module with its type (OO:ObjectType)

# save this textfile with name mystrops.nim
proc remNoneLetters*(s: string): string =
  result = newString(s.len)
  var pos = 0
  for c in s:
    if c in {'a' .. 'z', 'A' .. 'Z'}:
      result[pos] = c
      inc(pos)
  result.setLen(pos)

import mystrops

echo remNoneLetters("3h7.5g:8h")

const strfuncs = "stringutils"
import strfuncs

from mystrops import remNoneLetters

echo remNoneLetters("3h7.5g:8h")

from mystrops import nil

import strutils except toUpper

import tables as maps

from tables as maps import nil

# module gtkplus
import gintro/[glib, gobject, gtk4]
export glib, gobject, gtk4

Generally we try to arrange our own modules in a tree-like bottom-up structure. A module x may define basic types and simple functions working with these types, and a higher level module y may import all symbols from module x and extend the functionality. But in rare cases it may be necessary that the modules x and y import each other, as x has to use types or functions of module y, and vice versa. This case is called cyclic import and is currently not supported by Nim. Indeed we should generally try to avoid cyclic imports when possible, as cyclic imports make the software design difficult. But sometimes we can not really avoid these cycles. In that case currently the best solution is, to put all the concerned data types in a separate low level module, which is then imported from both other modules. The planned Nim version 2.0 may allow cyclic imports, so this restriction may vanish in the future.

// C program expecting one command line argument
// Compile with gcc t.c
#include <stdio.h>
int main( int argc, char *argv[] ) {
  printf("Executing program %s\n", argv[0]);
  if( argc == 2 ) {
     printf("The argument supplied is %s\n", argv[1]);
  }   else if( argc > 2 ) {
     printf("Too many arguments supplied.\n");
  }
  else {
     printf("One argument expected.\n");
  }
}

$ gcc t.c -o t
$ ./t Nim two
Executing program ./t
Too many arguments supplied.
$ ./t "Nim two"
Executing program ./a.out
The argument supplied is Nim two

from os import paramCount, paramStr

proc main =
  echo "Executing program ", paramStr(0)
  let argc = paramCount() + 1
  if argc == 2:
    echo "The argument supplied is ", paramStr(1)
    if paramStr(1) in ["-d", "--debug"]:
      echo "Running in debug mode"
  elif argc > 2:
    echo "Too many arguments supplied."
  else:
    echo "One argument expected."

main()

var s: string = stdin.readLine()
stdout.write(s)
stdout.flushFile

import json

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  l1 = Line(x1: 0, y1: 0, x2: 5, y2: 10)
  c1 = Circ(x0: 7, y0: 3, radius: 20)
  d1, d2: Data

d1.lines.add(l1)
d1.circs.add(c1)
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Circ(x0: 9, y0: 7, radius: 2))

let str1 = pretty(%* d1) # convert the content of variable d1 to a string
echo str1 # let us see how the strings looks
d2 = to(parseJson(str1), Data) # read the string back into a data instance
let str2 = pretty(%* d2) # and verify that we got back the original content
echo str2

# assert d1 == d2 would fail
assert str1 == str2

{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}

import json
from math import PI

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  d1, d2: Data

d1.lines.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.circs.add(Circ(x0: 7, y0: 3, radius: 20))
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

echo d1.circs[1] of Arc, " ", Arc(d1.circs[1]).endAngle

let str1 = pretty(%* d1)
d2 = to(parseJson(str1), Data)
let str2 = pretty(%* d2)
echo str2
echo d2.circs[1] of Arc

true 3.141592653589793
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
false

import json
from math import PI

type
  Line = ref object of RootRef
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    elements: seq[RootRef]

  Storage = object
    lines: seq[Line]
    circs: seq[Circ]
    arcs: seq[Arc]

const
  DataFileName = "MyJsonTest.json"

var
  d1, d2: Data
  storage1, storage2: Storage
  outFile, inFile: File

d1.elements.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.elements.add(Circ(x0: 7, y0: 3, radius: 20))
d1.elements.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.elements.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

for el in d1.elements:
  if el of Arc:
    storage1.arcs.add(Arc(el))
  elif el of Circ:
    storage1.circs.add(Circ(el))
  elif el of Line:
    storage1.lines.add(Line(el))
  else:
    assert(false)

let str1 = pretty(%* storage1)

if not open(outFile, DataFilename, fmWrite):
  echo "Could not open file for storing data"
  quit()
outFile.write(str1)
outFile.close

if not open(inFile, DataFilename, fmRead):
  echo "Could not open file for recovering data"
  quit()
let str2 = inFile.readAll()
inFile.close

assert str1 == str2

storage2 = to(parseJson(str2), Storage)

for el in storage2.lines:
  d2.elements.add(el)
for el in storage2.circs:
  d2.elements.add(el)
for el in storage2.arcs:
  d2.elements.add(el)

for el in d2.elements:
  if el of Arc:
    echo "found arc with endAngle: ", Arc(el).endAngle

For storing unstructured data Nim provides the io module with the File data type and related procs, and the streams module with the Stream data type and related procs. While a File in Nim is currently only a pointer to a C file, the streams module has a higher abstraction level. Although the Nim language does not directly support interfaces, the Stream data type of the streams module is some form of an interface, which is implemented by a StringStream and a FileStream data type. Internally this interface concept is realized by storing a set of function pointers in the Stream instance.

When we have to store unstructured data like text it is not always clear if we better should use Files or Streams. Streams may be the better choice when we (also) want to use a string as data source like a file or when we need the peek() functions of the streams module to access data without advancing the position in the stream.

We will use the File data type of the io module first. As the io module is part of the system module, we do not have to import it before we can use it. The principle usage of files is, that we call the function open() to open a file with given name, call some procs to write or read data, and finally close() the file. While Nim support destructors when we compile with --gc:arc or --gc:orc, the io module does not yet use them, so we should actually call close() to close the file.

Running that program will create a text file with this content in the current working directory:

At the start of our function we check if a file with that name already exists in the current working directory by using the function os.fileExists() to ensure that we do not overwrite important data.

Module io provides multiple overloaded open() procs. We use here a variant which returns a file, and raises an exception for the unlikely case of an error. We provide a file name and a file mode as parameters. We use mode fmWrite as we want to create a new file. Note that fmWrite would clear the content of an existing file, so we can not use fmWrite to append data to an existing file. We would have to use fmReadWriteExisting or fmAppend to append data to an already existing file. As this open() proc can raise an exception, it may make sense to enclose it in a try/except block, or we could use a open() variant which returns a boolean value to indicate success instead. When the file is successfully opened, we can use procs like write() or writeLine() to write text strings to the file. Both procs accept multiple arguments and apply the stringify operator $ on them before writing the content. WriteLine() writes a '\n' after the last argument to start a new line. When done we call close() to close the file. The operating system would close the file for us when our program terminates, so calling close is not that important, but when we open many files without closing them we may get errors from the operating system finally about too many open files and our program may fail or terminate.

The close() proc gets passed the file not as a var parameter, so it can not set the file to value nil. When the file has the value nil, then the close() call is ignored, but when we would call close() multiple times with a non nil argument we get a program crash. We may use the try/finally or the defer construct to ensure that we really close the file when done.

The io module provides some procs like writeBuffer(), writeBytes() or writeChars() which gives us as return value the actual number of bytes written. This return value should generally match the requested number of bytes to write, but can be smaller when the write operation fully or partially failed, e.g. because the storage medium had no capacity left.

When performance really matters, we should note that passing non-string arguments to write() or writLine() procs using their optional auto-stringify for us, involves allocation of new strings and cost some performance. When we have in our program already a string variable available, it can be faster to convert our data into that variable first and then pass that variable to the write() or writeLine() procs.

Reading strings from a file works very similar:

The readLine() proc reads in a line of text. The LF, CR or CRLF line end markers are not part of the returned text string. Of course we may get an empty string with length zero back when we read a line which immediately starts with LF, CR or CRLF, or we may get back a string with no visible characters but only a few spaces or tabulator characters '\t' when a line contains only white space. When our read() operations have moved the actual file io position to the end of the file, and we try to read more content, then an exception is raised.

The io module provides a readLine() proc that returns a newly allocated string, and one that takes an existing string as var parameter. The later may be a bit faster, as it can avoid the allocation of a new buffer when the passed string has already enough capacity.

The io module provides a function called endOfFile() with a boolean result which we can use to check if the end of file position is already reached. The provided functions readBuffer(), readBytes() or readChars() return the actual number of bytes read, which can be smaller than the requested value when the end of the file is reached earlier. Currently readChars() checks if the passed openArray[char] has enough capacity for the request, but readBytes() does no check!

We can use also the lines() iterator to iterate over the lines of a text file, or use the readLines() proc to read the content line by line.

As iterating over the whole file line by line moves the actual file position to the end of the file we called setFilePos() to move again to the start position. The readLines() proc takes a filename and the number of lines to read as parameters and returns a seq of strings. When the file does not contain at least the number of requested lines an EOF exception is raised. Another provided proc is readAll() which reads the whole file content into a returned string variable. For readAll() to work the actual file position has to be the the start of the file. In case of an error an exception is raised.

We can also write and read binary data directly to a file, without converting it to (human readable) strings first:

Of course these are low level, dangerous operations. While writeBuffer() should never crash our program, readBuffer() can do that easily when we specify wrong sizes or destination addresses, as that may overwrite other data unintentionally. So we would generally not use these procs directly but write more safe helper procs, when we really need or want this form of binary file access. Fast storing big data sets with restricted hardware may be an use case, e.g. storing a float32 takes only 4 bytes on the storage medium and file io is fast, while that number as human readable digits may need more than 8 bytes (1.234567E3) and converting to string and and parsing back costs some time.

In the same way we can use writeBuffer() and readBuffer() to store tuples, objects and arrays or sequences of these directly in binary form:

Output should look like

But of course this is dangerous and fragile. We just show that example as beginner generally ask about it, and may want to try it at least once. Obviously this can only work when the tuples or objects contain only plain data types, that is no string, no references and of course no other nested container types like sequences or tables. And reading back data may fail when we use a different OS or a different compiler version.

The io module provides the File variables stdin, stdout and stderr, which are the standard input, output and error streams. Sometimes we use stdout.write() instead of the common echo() proc when we want to write something to the terminal window without moving the cursor to the next line already.

An important function of the io module is flushFile(), which is used to ensure that all buffer content of buffered files is actually written to the file. This is important when we use the stdout File variable, maybe to ask the user a question in the terminal window. We would call sdtout.flushFile() to ensure that the user really sees the text on the screen immediately. The echo() proc calls flushFile() automatically after each output operation. When we close a file flushFile() should be called automatically, but when our program is terminated without calling close() it may depend on the actual implementation and operating system

The io module provides some more useful procedures, but we will stop this introducing section here and continue with the streams module in the next section.

References:

• https://nim-lang.org/docs/io.html

A stream is an abstract interface for performing certain I/O operations, which was introduced by languages like C or Modula-2 decades ago. The streams module of the Nim standard library provides a FileStream and a StringStream implementation, which behaves very similar. Nim’s streams module provides similar functions as the io module with its File data type, but it can operate on strings instead of on Files, and it provides a set of peek() functions to access data at the current read position without moving forward. And some functions are more robust, for example closing a stream multiple times does not crash the program, as the first close() call sets the file variable of file streams to nil, so that following close() calls are ignored. Currently the streams module does not support automatically closing of streams when they go out of scope.

We can create a new FileStream by calling the overloaded procs newFileStream() with an already opened file or a filename as parameter, or we can use openFileStream(). The later raises an exception when the stream can not be opened, while the former procs just return nil. We can write and read textual data with the streams module in a very similar way as we did it with the io module and the File data type:

We test again if a file with that name already exists. Then we try to create a new FileStream by using file mode fmReadWrite, so that we can write and read from that file. Finally we write two numbers, which are automatically converted to strings, set the file position back to the beginning and verify what we wrote by reading it in again, before we close the stream.

In a very similar way we can write to and read from string streams

In the example above we do not test if the stream variable is not nil, as newStringStream() should never fail.

For buffered streams we can call flush() to ensure that the buffer content (of file streams) is written, similar as we can do it for plain Files of the io module. Instead of io.endofFile() we use the proc atEnd() to test if the current stream position is already at the end of the stream. Functions getPosition() and setPosition() are available to query or set the actual position in the stream. While the io module with its File data type supports for io.setFilePos() although position modes relative to the actual position or relative to the file end, streams.setPosition() use always absolute values, that is positions measured from the beginning of the stream. The streams module provides also the low level procs readData() and readDataStr() which reads data to a memory region or into a string and returns the actual number of bytes read to indicate success. And as for the io module a proc readAll() is available to read all data of a stream into the returned string variable.

The procs writeLine() writes the passed arguments always as strings. The overloaded write() procs with varargs arguments write the passed values as strings and apply the stringify operator $ if necessary. The same does the writeLine() proc, but it writes a newline character when all passed variables have been written. One more overloaded write proc for single string parameters exist.

But for single non string arguments a generic write() proc is used, which writes numbers (and other data types like boolean types or single characters) directly in binary form without converting them to strings.

To read the binary numbers back we can use functions like readFloat64() which have a well defined return type and read a fixed number of bytes. Or we can use the generic read() proc which accepts a var parameter which defines the data type that we intend to read in binary form. Additional to the various read() procs the streams module provides a set of peek() procs which reads data in without moving the actual position in the stream forward. This may be useful for parsing of files, as we can read the same information multiple times easily. Internally the peek() functions uses a call of setPosition() to save the current position and one more call of setPosition() to set back the old position to the initial value, so peek() has some overhead.

In the example above we write a three byte long string and a float64 to the file stream. We call setPosition(0) to read the stream from the beginning again, and then read in an int16 with the function peekint16() without moving the actual position forward, followed from readInt8(), which moves the actual position one byte forward. (Instead of readInt8() we could also call read() with variable i8 as passed var parameter.) Then we read in two bytes and finally the float64 value at the end of the stream. Finally we check by use of the function getFileSize() from the os module if the file has really the expected size.

The streams module provides many functions, and the possible writing data as strings or in binary form can make using that module a bit daunting at first. But most procs have examples in the API docs which helps you using it.

References:

• https://nim-lang.org/docs/streams.html

We discussed Nim’s string data type already in part II of the book. Remember that a string in Nim is a variable size container for ASCII characters. strings can be plain ASCII strings, or the bytes of the string can be interpreted as unicode glyphs. Nim has also a cstring data type, which was initially introduced to be compatible with the character arrays used in C libraries as strings, but is now called compatible string as cstrings do also support the JavaScript backend. Nim strings can be passed directly to C libs, as a Nim string contains a Null terminated buffer for the actual data, which is identical to a C language string. So whenever we convert a Nim string to a C language string or pass a Nim string to a C library, this is free of costs, while converting a C string to a Nim string always means allocating a new Nim string and coping the data content. Technically for a Nim string s addr s[0] is the C string pointer, called * char in C language. Whenever we pass strings to C libraries we have to care for the fact that Nim’s garbage collector may deallocate the string automatically. Most C libs create copies of passed strings when the libs use the string for a longer time span. GTK for example does this with text for its widgets. But when the C lib does not copy the string but use it directly for a longer time, then it can occur that the Nim code frees the string, as the only one Nim variable referring to the string goes out of scope, but the C library still uses the string. For that rare case we may call GC_ref() on the string to prevent garbage collection, but that may generate memory leaks then. For the case that C libs create strings, they provide generally also a function to deallocate the string. When we use such a C function, it is generally the best solution that we copy the string from the C lib to a Nim string and immediately deallocate the C string by a call of the provided free()/dealloc() function. For most C libs there exist good high level bindings which do not have this issues, so we generally can use the C libs like pure Nim libs.

Nim' s system module provides already some basic string operation like accessing single characters by the subscript operator [], accessing slices of multiple adjacent characters, or joining multiple strings with the & operator. The overloaded add() functions to append single characters or other strings to existing string variables are also provided by the system module.

We start above example by assigning to the string variable s a string literal, then we append one more string literal, and finally replace the last character and the first five characters by another character and by another string. Note that by using the slice operator we can not only replace character ranges, but we can also replace slices of different length. This way we can also delete ranges in the string by replacing it with the empty string "".

The system module also defines the stringify operator $, which converts expressions to the string presentation when we put it in from of it. procs like echo() apply the stringify operator automatically on all of its arguments when necessary. And the system module provides the contains() function which we can use to test if a string contains a character. Instead of contains we can also use the in operator.

The system module also provides the procs newString() and newStringOfCap() which are mostly used for optimizing purposes. The function newString(n) creates a string of length n, but with uninitialized content. We would have to assign characters to the positions 0 .. n-1 to create a valid string. Function newStringOfCap() creates a string with length zero, but with a buffer capacity of n characters. When we know the needed buffer capacity, or at least a lower bound of it, it makes sense to create the string with newStringOfCap() with optimal buffer size to avoid reallocations. Of course we could still append more data, Nim would allocate a larger buffer and copy content.

Filling in characters into an existing string by use of the subscript operator [] is faster than appending single characters with the add() function, because the add() function has to check if the string has still enough capacity and because add() has to increase the actual string length by one for each call.

Note that a single character like 'a' is very different from a string with only one character like "a". A character in Nim is nothing more than a single byte, while a string — even one with only one character or an empty one — is an opaque entity with length, capacity and a pointer to a data buffer. When a single character is sufficient, we should use that and not a string containing a single character. A function call like s.add("a") may produce less optimized code than s.add('a'), but maybe the compiler optimized the former for us. When we consider optimization we may wonder if in

line 3 allocates a new string or just copies the string literal "Bye." in the existing data area. Well we would hope for the later of course.

Another interesting question is if in

we should better use line 3 or line 4. Line 3 looks more clear and we assume that it also would produce better code, as the actual append operation is avoided.

Often used functions are len() and setLen() to query and set the length of a string. While len() looks like a function call, it is a compiler intern function, so calls are fully optimized. So it is OK to write

It is not necessary to introduce a temporary variable like let l = s.len() to avoid many function calls. In C this is different, as in C a call of function strLen() would not only imply a function call, but that function would really have to count all characters up to the terminating '\x0' as C strings have no length field. The function setLen() is generally used to truncate strings. A call like s.setLen(0) makes s look like a newly allocated string, but its data buffer can be reused. Reusing strings is generally better for performance than allocating many new strings. The setLen() function is rarely used to increase the string length — in that case an allocation of a larger data buffer can occur, and the string would still look the same as initially all the new string positions would still contain the default binary zero content. We would have to fill in actual characters by use of the [] subscript operator. To get the lowest and highest character index of a string we can use s.low and s.high. s.low should be always return zero, and high() is identical to len() - 1, so high() is -1 for an empty string. Note that while calling high() and len() on a Nim string has no costs, this may be different to C strings as these have no length field.

The system module provides the overloaded & operator which we can use to concatenate chars and strings, the &= operator to append a new string to an existing string, and the add() functions to append characters or strings to an existing string. For best performance we should try to use always the most simple, "native" operations, at least as that does not make the code ugly or we know for sure that the compiler optimizes it for us.

The function add() can be also used to add a cstring to a string, and the JS backend allows even to append one cstring to another one by add() calls.

Module system also exports a substr() proc which copies and return a slice of a string. Overloads with optional first index with default 0 and optional last index with default s.high exists.

And of course == and != operators for string comparison are provided. To test if a string is empty we can compare with an empty string literal or test if len() is zero. The later is guaranteed to have best performance, but the former is a bit shorter and should be not bad performance wise. Some other languages provide an empty() test function for this, we may define our own when we really want.

When we pass a string to a proc that does no operations with it, maybe it only calls echo() or stdout.write() to print it, then it may have a tiny performance advantage to pass it as cstring. This is similar as we may pass sequences as openArray to functions, which also avoids one level of indirection. Also note, that while a Nim string is a value type, so we can not test it for nil or return nil from a proc that shall return a string, this restriction does not hold for cstrings.

Module stringutils provides a set of functions (120 currently) and a few iterators for simple string operations. Using that functions and iterators is simple in most cases and generally well explained in the API docs. Remembering which functions exists, their exact name and the function arguments can be a bit difficult at first. We will introduce in this section some of the most used or more difficult functions, and give some warnings when the actual performance may be not as good as expected.

Performance critical operations are generally that one which has to allocate new strings or that has to shift many characters, like text inserting operations. Note that some functions of this module like toUpperAscii() work only with the lower and upper ASCII letters. For unicode operations we may need the unicode module.

The stringutils module support string interpolation by use of the % operator.

We can use $1 up to $9 to mark positions where string n from the array should be inserted, or just $ to insert the strings in the order as they appear in the array. We can also use named insert markers and specify name-value string pairs in the array. For a single string we can omit the array and pass just as string, and finally we can use the format() proc to enable stringify for the parameters.

We mentioned already the useful but performance critical set of split() functions:

We used the split() variant which accepts a string as split marker. This function accepts one optional parameter for the number of splits to execute and returns a sequence containing the single strings. The split marker string, also called separator, is removed from the strings. The default value for the number of splits is -1 indicating that we want a split at each separator position. If we specify a positive number n, then only n splits are executed and the last element of the returned sequence will contain the remainder of the string. When we specify for the intended splits a value which is larger than the number of contained split markers, then we get a full split.

The reason why this function is not very fast is that it has to allocate a sequence for the return value and a new string for each split and one more for the last string or the remainder. For the case that we do need only the first few strings of the split, it is a good idea to specify the number of actual splits to increase performance.

The strutils module provides overloaded functions which use single characters as separators or which accepts a set of characters as separators, so we may split at space, tabulator, comma or semicolon with {' ', '\t', ',', ';'}. And a function splitWhitespace() is available to split at whitespace like spaces and tabulators, which removes all the whitespace between the strings. Notice that the split() function for strings or single characters does one split for each separator, so we can get empty strings as result as in

An interesting behavior of splitWhiteSpace() is that whitespace at the start or end of a string is just ignored, while the split() function returns additional empty strings when the string to split starts or ends with the separator:

Another function is splitLines() which splits a string at CR, LF or CR-LF characters.

For these splitting functions also iterator variants exists, which behave like the functions with same name. When we limit for the iterators the number of splits to perform, we may get as last returned value the remained string. You may consult the API docs for details if necessary.

Functions with names rsplit() are also available which behave like split() but start the splitting process from the end of the string, so we can get the file extension of a file name with something like filename.rsplit(1)[^1].

The functions removePrefix() and removeSuffix() can sometimes help to avoid expensive split operations. There is an overloaded function that removes all single characters, all characters from a set or a single string:

Other useful functions are startsWith() and endsWith() which accept a single character or a string and return a boolean value:

An efficient function similar to find() is continuesWith() used like "Nim Language".continuesWith("Lang", 4) which tests is a string contains a substring at position n.

We can use the join() proc to join strings in arrays or sequences to single strings with an optional glue string. Join() works also when the elements are not already strings — in that case the stringify operator is applied first:

The overloaded find() functions accepts optional start and end positions and returns the index position of the first match or -1 when the search gave no result:

The contains() function can be used to test if a character, a set of characters or a substring in contained in a string. Instead of contains(s, sub) we can write sub in s. Note that a function variant for single characters is defined in the system module.

The replace() function can be used to replace all occurrences of a character or a substring in a string and to return the new string. We can use replace() with an empty replacement to delete a substring. The also available delete() function is used to delete a range specified by two indices in place.

Additional a replaceWord() function exists, which does only replaces whole words, i.e. words that are surrounded by word boundary characters like spaces, tabulators or newlines.

The function multiReplace() is a variant that can replace multiple substring/replacement pairs passed as tuples in one pass.

The function strip() can be used to remove multiple characters from a set at the start and the end of a string. The default character set is whitespace, and per default strip removes characters at both ends of the string. The function stripLineEnd() removes a single line end marker as \r, \n, \r\n, \f, \v from the end of the string, but only once.

Sometimes useful is the boolean function isEmptyOrWhitespace() which checks if a string is empty or contains only whitespace. Also useful can be the function repeat() which returns a string that contains the passed character or the passed string n times, and the function spaces() which returns a string containing only n spaces.

For single character tests we have functions like isDigit(), isUpperAscii(), isLowerAscii(), isSpaceAscii, isAlphaNumeric(). Function isDigit() test for characters '0'..'9', isUpperAscii() for 'A'..'Z', islowerAscii() for 'a'..'z', isSpaceAsccii() for ASCII whitespace (' ', '\t') and isAlphaNumeric() test for lower or upper case ASCII letter or a decimal digit.

Function toLowerAscii() convert all the characters 'A'..'Z' to lower case, and toUpperAscii() converts all the characters 'a'..'z' to upper case. The function argument can be a string or just a single character. With capitalizeAscii() we can convert the first ASCII character of a string to upper case.

The overloaded functions count() can be used to count the characters or substrings in a string, and countLines() is available to count the number of lines, where lines are separated by CR, LF or CR-LF.

Sometimes we may also need functions like formatFloat(), formatBiggestFloat() or formatEng() to format float numbers for output purposes. You would have to consult the strutils API docs for all the format details. An intToStr() function with an argument to specify the minimal string length is also available. The string may get leading zeros for alignment.

Finally some important functions are the parsing functions like parseFloat() or parseInt() which converts strings to float or integer numbers. Both raises an exception when the string does not contain a valid number.

The strutils module contains some more not that often used function, like functions to convert data to hexadecimal, octal or binary representation, or to parse numbers back from that string representation into numbers. Other functions like align(), center() and indent() are available for string positioning. We will not try to describe these seldom used functions here, as it is hard to remember the detailed behavior. You should skim the API docs and consult them when you need one of the exotic functions or when you have forgotten how to use a concrete function.[37]

The module parseutils provides a set of functions for efficient and fast parsing of strings. The functions avoids the allocation of new strings by passing back results in var string parameters and by returning the number of processed characters. The module parseutils is a good choice when we need efficient parsing of strings and the input strings have a simple structure. For more complicated input data we may have to use RegEx or PEGs. Let us assume that we have a set of library names which includes the version numbers, but we need the plain names. The function parseWhile() is a good candidate for this task:

First we allocate a string with enough capacity, so that the parse() functions can use it without having to do allocations. As we want to receive the plain names, using parseWhile() with a charset as last parameter may be a possible solution. But as we see this will not really work for webkit2gtk which contains a digit in its name:

We can fix this by passing the extended char set {'a'..'z', '0'..'9'} to parseWhile() or by use of parseUntil() with a character set that does not belong to a name. Both functions return the number of processed characters and provide the captured string in the passed var parameter. Not the we can use the slice operator .. to specify character ranges for the charset parameter when the characters build a continues sequence in the ASCII table.

A related function is skipUntil(), which we may use when we are more interested in the version numbers after the name:

All these functions accept an optional start parameter as last argument. A common use case is to use an integer position variable initialized with zero, which we increase by the returned value so that the parsing can continue at the current position in the string. The next example will use this strategy. For parseUntil() overloaded functions are available which gets not a char set but a single character or a substring as parameter. These functions stops parsing when the character or the substring is found and return that position.

Functions like parseInt() and parseFloat() can be used to extract numbers from strings:

In above example we used the module prefix as strutils contains also a parseInt() and a parseFloat() function. The functions parseBin(), parseOct() and parseHex() behave in a similar way. Returned is the number of processed characters. We add the returned value to the start position so that parsing can continue at the new position.

There are some more functions available in this module, which we will not discuss further. It is enough that you know that this module exists and provides some efficient parsing functions. Whenever you should really need one of these procs you would have to consult the API documentation for details.

The strscans module provides a scanf() macro which can be used to extract substrings from textual user input. The content of the substrings is automatically converted to Nim variables of matching data types.

Processing of well defined strings is easy, and with user defined matcher functions even text input with less strict shape can be processed.

Let us start with a simple example: We may have to create a program where the user should be able to create rectangles by entering the coordinates of two opposite corners in the form

The first parameter for the scanf() macro is the user input string, and the second parameter is a pattern string that specifies how the input string should be processed. The following parameters are variables that gets the results of the input evaluation. The second parameter has some similarity with a regular expression. The letters after the dollar sign specifies the data type of the substrings, $i or $f ask to process an integer or a floating point number, and $w requests to process an ASCII identifier. Other characters are captured verbatim, that is the space character after $w has to match a space in the input string, and the comma characters that separates the $f has to match commas in the input string. The scanf() macro supports capturing of some more data types, i.e. $c for an arbitrary character or $s for optional white space. The optional white space is not captured, just ignored. With the use of $s our program allows already a more flexible input string:

To allow to process even more flexible input strings it is possible to use user definable matchers in form of Nim procs with a well defined parameter signature. There are two different types of matcher procs supported — matchers to just skip a part of the input string, and capturing matchers. For the next example we will use a proc which can skip various separators like comma, semicolon or white space. And we will use a capturing matcher proc for the object name.

The use of our user definable matcher sep() allows to separate the four numbers with a colon or a semicolon with arbitrary leading or trailing white space, or with only white space. Multiple colons or semicolons between two numbers resulting from a typo would be not permitted.

The signature for this matcher proc has this shape:

The first parameter is the string to process, and the second parameter is the position at which the processing should start. (Or in other words, the second parameter of integer type is the actual position in the input string, that position moves during the whole capering process from the start of the input string to its end. The end may not be reached if the capering fails at some point.) The last parameter has always the type set[char] with a default value indication which characters that proc can process. Actually a default value seems to be necessary, but the actual value seems not to matter. The proc returns the number of characters that should be skipped. Zero is a valid return value, so we can support optional separators. In most cases separators are necessary to process the input string, but we can imagine input formats where separators are optional, e.g. when an integer number is followed by a name. A name never starts with a digit, so the boundary between the two values is well defined. This none capturing matcher proc is called by use of $[sep] in the pattern string.

The signature of the capturing matcher proc has this shape:

That proc also gets as parameters the input string and the start position, and has to return the number of processed characters. But additional a var parameter of arbitrary data type is used to return the result of the capture, and the last parameter with arbitrary data type can influence the capturing process. One possible use of the last parameter is to use an integer value to limit the maximum number of characters to process. This proc is called in the pattern string by using curly braces like ${stt(0)}.

Scanf() returns true when all the parameters match, for that case all the passed in variables get assigned a value. Currently scanf() does not support the capturing of optionally data, as the whole processing stops when one capture fails, i.e. when $i is used to request the capture of an integer value but the input string does not contain decimal digits at the current capture position. In the same way the whole capturing process stops when a user defined capturing matcher returns zero as no capturing is possible. So intermediate optional arguments are currently not supported. When the processing stops due to missing arguments, scanf() returns false, but the already processed captures still have a valid value assigned. In this way we can use at least optional trailing arguments.

As next example for the use of the scanf() macro we will give a real world example: A simple CAD (Computer Aided Design) program has a PCB (Printed Circuit Board) mode, in which the user can create new PCB pads by entering the pad data in a text entry widget. A PCB pad is a rectangular shaped copper field which has an associated number and a name and maybe rounded corners. The user should be able to enter two 2D coordinates, the corner radius, an optional x/y translation for the next pad of same size, followed by the number of pads to create and the pad number and name. That is

The first five arguments are mandatory, the rest is optional with default values. The user should be able to separate the arguments with white space or with a colon or a semicolon. Additional the values x2 and y2 can be preceded with a + character to indicate that the x2, y2 tuple is not an absolute coordinate value but the width and high of the pad.

A program fragment to process this form of user input may look like

When we compile and run this program we get this output:

The first scanf() call uses the sequence $[sep]${pls(0)} which fails when the float value has no leading + sign, so this call is of no real use. The second and third plus() proc processes the separators as well as the optional + character, so that proc has never to return zero, and the capturing process continues. For the last scanf() call we give as input only five values, so scanf() returns false, but the first five values gets assigned values, and the rest has default values. One restriction of above code is, that we have always to start the input string with pad, otherwise the processing stops immediately. As scanf() does not support the capture of boolean values, we uses the integer data type for the variables px2 and py2. The value zero means that there is no + prefix, and 1 indicates that there is a plus prefix.[38]

The next tiny example shows how we can use the last parameter of the user defined matcher proc to control the matching process:

We want to capture two integer values, each with one or two decimal digits. By passing the upper limit of digits to the ndigits() proc, we get the intended result even when the user does not separate the two numbers with white space. Additional we have used $. at the end of the pattern string. The $. matches only when the end of the input string is reached, so that scanf() would return false if there are more characters in the input string left.

The latest version of the strscans module provides also a variant of the scanf() macro called scanTuple() which returns a tuple. We could use it in this way in our example above:

So we have not to declare the capering variables in advance. The final result of the scan process is returned in an additional boolean variable. When we use user defined matchers as above, we have to specify the data types of the returned values as additional parameters after the pattern string.

Additional the strscans module provides a scanp() macro which works somewhat similar as PEG or RegEx libraries. We will not try to explain the scanp() macro, as its use may be too difficult for a beginner book. And when we really have to process text strings with regular expression grammars, then we can use the available RegEx or PEG modules which have no restrictions and work for Nim in a similar way as for other programming languages. We will introduce the Nim RegEx and PEG modules later in the book — maybe we will compare the scanp() macro there.

With the fmt() macro from the strformat module we can format and interpolate strings similar as Python3 with its f-strings.

From the four ways to print some text the last one with fmt() is the shortest and maybe the cleanest. As fmt() is a macro which is processed at compile time there is no unnecessary run-time overhead involved. A small restriction of fmt() is, that it’s argument is regarded as a generalized raw string literal. So we can not use escape sequences like "\n" in the string literal. But the strformat API docs mention various solutions for this: We can use the unary & operator instead of the fmt() call, or we can use the notations {'\n'}, fmt() or "".fmt.

The fmt() macro works also with multi-line raw strings:

The fmt() macro accept a few directives for the formatting of integer and floating point numbers like

The basic pattern is, that the numeric variable is followed by a colon and the total number of desired characters. We can precede the number with a < or > to indicate left or right alignment, and for floating point numbers we can use a notation similar as used for printf() in the C language: n.mf stands for a float formatted with n characters total and m decimal places. As in C we can use e instead of f to indicate scientific notation. If the value which specifies the total number of characters starts with a zero digit, then the formatted number uses zeros instead spaces for leading digits. And X after the colon generates hexadecimal value for integer numbers.

A useful property of the fmt() macro is, that we can put an equal sign into the curly braces to get the initial expression both as string and as interpolated value, as in

This is similar as the dump() macro from the sugar module and is mostly used for debugging purposes.

To use curly braces as literals in a fmt() argument we can use character literals with a backslash as we did to include a newline character, or we can use an extended fmt() macro with two additional arguments which specifies the two characters that should be used instead of {} to mark the expression which should be interpolated:

We will not try to explain all these various formatting options in detail, as it is really hard to remember. It is enough that you know that these options exists, so you can consult the API docs for details when you would need it.

References:

var s0: seq[int]
var s1: seq[int] = newSeqOfCap[int](2)
var s2: seq[int] = newSeq[int](2)

s0.add(3)
s0.add(5)
s1.add(3)
s1.add(5)
s2[0] = 3
s2[1] = 5
echo s0; echo s1; echo s2 # @[3, 5] for each

s0.add([7, 9])
s0 &= s1
s1 = s0 & s2
echo s1 # @[3, 5, 7, 9, 3, 5, 3, 5]

var i = 7
var j = 9
var s0 = @[1, 2]
s0 = s0 & @[i, j] # don't use this variant!
s0.add([i, j]) # faster
echo s0 # @[1, 2, 7, 9, 7, 9]

var s0 = @[1, 3, 5]
var s1 = s0[1 .. 2] # @[3, 5]
for el in s0[1 .. 2]: # may create a copy of the seq
  echo el

This module defines some useful procs, iterators and templates for working with arrays and sequences. Some functions of module sequtils use an generic openArray parameter and so can be used for strings as well. While the max() and min() procs are available from the system module, the minIndex() and maxIndex() procs are provided by sequtils:

Sometimes we may need a minmax() proc which gives us both values, but that one is currently not available. We have to create it our self if needed, when performnce is not that critical we can call min() and max() separately.[39]

The functions minIndex() and maxIndex() as well as count() or deduplicate() can work with strings also:

The function name deduplicate() may be irritating, as the function does not work in place, we may expect the name deduplicated() as for sort() and sorted(). Other programming languages use the name uniq() instead. Deduplication is easy when the elements are sorted, as for that case we can just iterate over the seq and ignore equal adjacent items. That is why deduplicate() accepts an optional boolean parameter indicating if the seq is sorted. If the seq is not sorted, it may be necessary to create a temporary set to store the already seen items so that they can be ignored the next time when they occur again in the seq.

The function concat() can join multiple sequences (yes only sequences, it does not work with arrays currently), maybe that is more efficient than using the & operator. And insert() can insert a new value at a position by shifting following items, and delete() allows to remove a range of items from the seq when we specify two index positions:

Proc repeat() is used to create a seq which contains a single value multiple times, and cycle() repeats the items of an existing seq multiple times:

A bit more complicated but really useful are functions like map(), filter(), keep() and the corresponding …​It() templates:

The map variants return a new seq, with an operation performed on all items. The returned seq can have a different base type. In line 4 we used the ⇒ operator from the sugar module for a simpler notation

The filter() variants apply a proc with boolean result type on the seq items and return the items for which the result is true. Remembering if the elements for which the procs gives a true result are returned or removed from the initial seq may be not easy. It helps to remember that filter() behaves like the keepIf() procs — items with positive result survive.

In above code we used toSeq() with a slice argument to create an initial sequence with continues integers from the slice. The apply() function performs a transformation operation on all the items, and keepIf() preserve only the items for which a boolean predicate evaluates to true.

Two useful predicate functions are any() and all() to check if at least one item fulfills a condition or if all items fulfill a condition:

With zip() we can join the items of two sequences to tuples, and with unzip() we can separate the tuple items again in two separate sequences:

Finally sometimes the templates foldl() and foldr() can be useful for folding a sequence, that is to generate one final value from all the items. The fold templates uses the variables a and b to generate the result, a + b would sum all the items. Foldl() performs the operation from left to right, returning the accumulation and accepts an optional start value, while foldr() starts from right, i.e. with the item at the end of the seq.

The sequtils module contains some more procs, templates and macros that are not needed that often. It would not make much sense to mention all of them here, as it is already not easy to remember the ones that we have introduced above. You should skim the sequtils API docs from time to time to remember what is available.

Maybe you have missed the difference of two sequences which some other programming languages provide:

Well that is an expensive operation, O(n^2) if implemented in a naive way, as it may iterate for all items in the second seq over the whole first seq to remove the unwanted items. A better approach is to convert the items in the second seq to a temporary (hash) set to allow faster query:

A fast on the fly solution is this, as suggested by someone in the Nim forum:

When we should need this operation often, we may define our own proc like

Note that this preserves the order in the first seq, which is often requested. If order is not important, then we could convert both sequences to a set and build the set difference — but when order does not matter we may use sets instead of sequences from the beginning.

Maybe you missed also a shift() function which other container types or programming languages may provide, that is used similar to pop(), but deletes and returns the the first item of a seq? Well it should be obvious why a shift() is not provided by default and why such a function would generally be avoided — the function name gives you already a good hint. And if you should need such a function it should be no problem to implement one if efficiency is really not important. But maybe in that case it would be better to use a different container type, maybe the double-ended queue provided by the deques module.

References:

type
when defined(js):
  type Ui = uint32
  const randMax = 4_294_967_295u32
else:
  type Ui = uint64

  Rand* = object # State of a random number generator.
    a0, a1: Ui

proc rotl(x, k: Ui): Ui =
  result = (x shl k) or (x shr (Ui(64) - k))

proc next*(r: var Rand): uint64 =
  let s0 = r.a0
  var s1 = r.a1
  result = s0 + s1
  s1 = s1 xor s0
  r.a0 = rotl(s0, 55) xor s1 xor (s1 shl 14) # a, b
  r.a1 = rotl(s1, 36) # c

from times import getTime, toUnix, nanosecond
import random

let now = getTime()
var rstate = initRand(now.toUnix * 1_000_000_000 + now.nanosecond)

for i in 0 .. 5:
  echo rstate.rand(5) + 1 # dice roll

for i in 0 .. 2:
  echo rstate.rand(100.0) # float random number in range 0.0 .. 100.0

import std/[random, times, monotimes]
from std/math import sqrt

proc warmup =
  var x: float
  for i in 1 .. 1e10.int:
    x += 1.0 / i.float
  echo x

proc main1 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = cpuTime()
  let res = sqrt(x) + y
  let stop = cpuTime()
  echo stop - start
  echo res

proc main2 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = getMonoTime()
  let res = sqrt(x) + y
  let stop = getMonoTime()
  echo stop - start
  echo res

randomize()
warmup()
main1()
main2()

$ ./t1
23.6030665949975
4.98999998654881e-07
4.676567857718999
42 nanoseconds
8.527354349689125

import std/[random, times, monotimes]
from std/math import sqrt

proc main3 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = getMonoTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = getMonoTime()
  echo stop - start
  echo res

proc main4 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = cpuTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = cpuTime()
  echo stop - start
  echo res

randomize()
main3()
main4()

$ ./t2
112 microseconds and 701 nanoseconds
436909.89897942
0.0001135580000000001 # 114 microseconds
437222.3001038401

type
  Customer = object
    lastName: string
    firstName: string
    age: int
    postalAdress: string
    phone: string
    credit: float

customers: seq[Customer]

# ...

var found = false
for c in customers:
  if c.lastName == queryName:
    found = true
    echo "Person ", queryName, " has a credit limit of ", c.credit
if not found:
  echo queryName,  "not found"

intVal = uint(s[0]) * 256^0 + uint(s[1]) * 256^1 + uint(s[2]) * 256^2 + ...

import std/tables

type
  Customer = object
    lastName: string
    firstName: string
    yearOfBirth: int
    postalAdress: string
    phone: string
    credit: float

var customers: Table[string, Customer]

proc addNewCustomers =
  var c: Customer
  c = Customer(lastName: "Turing", firstName: "Alan")
  c.postalAdress = "England"
  c.yearOfBirth = 1912
  customers["Turing, Alan"] = c

  c = Customer(lastName: "Zuse", firstName: "Konrad")
  c.postalAdress = "Germany"
  c.yearOfBirth = 1910
  customers["Zuse, Konrad"] = c

proc queryCustomer(key: string) =
  if customers.hasKey(key):
    echo "known customer:"
    echo customers[key]
  else:
    echo "customer key not found in data base"

addNewCustomers()

queryCustomer("Zuse, Konrad")
queryCustomer("Gates, Bill")

let dummy = Customer()
let query = customers.getOrDefault(name, dummy)
if query.lastName.len == 0: # we know all entries in the database have a lastName, so we got the dummy default value
  echo name, "not found"
else:
  process(query)

import tables
var ct: CountTable[string]
ct.inc("Nim")
ct.inc("Rust")
ct.inc("Nim")

echo ct["Nim"]
for k, v in ct:
  echo k, ": ", v

The tables module uses the hashes module to calculate the hash value for the keys that we use to access table content. For many data types the hashes module already defines a hash function. When we would like to use tuples or object data types as keys for table access, then we would have to define a hash function for that key objects first. The API documentation of the tables module contains an example for this, where as key an object data type with firstName and lastName fields is used to store salary entries in the table. While firstName and lastName are strings, and for single strings a predefined hash function is available, we have to declare another hash function for objects with two strings:

The hash generation is a bit cryptic: First we mix various existing hash values using the !& operator, and finally we use the !$ operator to generate the final hash value. For details please see the API documentation of the hashes module.

Hash tables can be seen as a way to attach arbitrary data to other data. The above example attaches a "salary" to a person object. In most cases we would just create one more object field when we have to store more data, but sometimes that is not easily possible. One example is when we use a low level C library, which gives us some C objects back. Maybe we use an advanced C or C++ math library like CGAL, and we get some abstract low level objects from it, maybe circles with center coordinate and diameter. As that objects are not Nim object, but C or C++ entities, we can not easily subclass them to attach more properties like a color attribute. But as each entity has a unique address, we can just use a table with key type address and all the needed values like color as data. That would be some overhead of course, as each color lookup would mean a table access, but it is a simple solution.

We can even attach properties to plain data types this way:

For floats a predefined hash function is available, so the code above should work. But floats as keys are a bit fragile due to the fact that float math is not really exact. So the last line in above code may raise an exception due to access of a non existent entry, as the difference 5.0 - 3.0 may not exactly be identical to the value 2.0.

Hash tables can be even useful containers, when we already have numeric data as possible keys for indices in a sequence: In mathematics we could have a two dimensional array, that is an array of array in Nim, to store matrices. This is OK when the matrix is really populated, that is most entries have meaningful non trivial values. But for very large, loosely populated matrices, with mostly just zero or one entries, storing the whole matrix as a table using a row, column tuple as key, may save a lot of RAM.

When we use objects or references to objects as keys for tables, we have to remember how Nim compares value and reference types:

The output of above program is

By default the == operator compares content for value objects, but the instance addresses for references. Because of this it makes sense to define hash functions for object types and ref object types in a compatible way: We use the hash value of the single integer field of our value object as hash result for the whole object, and we use the address of the instance for the hash value of the reference object. As different instances of ref objects have always different addresses, the hasKey() does return false when we use as argument a different instance variable, independent on the content of its fields.

For special use cases we may redefine the == operator, but we have to ensure that the defined hash function matches the == operator: When a == b is true, then hash(a) has to be identical to hash(b)! The reason is that tables first compare the hash value of query key with key of entities in the table, and only for matching hash value do comparison of the actual data content.

Hash table lookup is fast. We say that hash table lookup is a O(1) operation, which shall indicate that the time needed for doing a table lookup does not depend on the total number of entries stored in the table. The reason for that is that for a lookup it is necessary to calculate the hash value, do the modulo operation and access the table content and maybe a few of the following table entries in the case that the first entry is not a match. Storing data is also an O(1) operation, as it works very similar, as long as the table is not already too dense populated so that a recreation is necessary. In that case that single storing operation is obviously very slow, but that occurs very rarely, and maybe not at all when we use a large enough table from the beginning. Still small tables are much faster than larger tables due to cache effects. For small tables all data may fit into the caches of the CPU, while for large tables most data is located outside of caches in RAM, and RAM access is magnitudes slower than cache access.

And hash table lookup is slower than array or seq access. To access an element from an array or seq we have only to multiply the index value with the byte size of the stored elements type and maybe to add an offset when the array does not start at index 0. For tables we have to calculate the hash value, do the modulo operation, access some elements at the calculated position, and most importantly, to compare the content at that positions with the actual key data. If the key is a string, a few string comparisons (at least one) are necessary to determine if the query element is available in the table. So while array access may take less than a nanosecond on modern hardware, table lookup may take a few dozens of nanoseconds. Lookup with string keys is generally slower than for other key types likes integer, as for string comparison it may be necessary to compare many characters to get a result and because strings generates some memory indirection by the fact that string content is stored somewhere in the heap outside of any cache.

At the end of our introduction to hash tables we will present a very useful, but maybe not that obvious property of hash tables: The keys used for table access don’t have to be simple data types, but can be container types like tuples or arrays. Imagine you have a map in 2d with a set of points on that map each presented by an x, y coordinate pair. That points could be cities, and some cities may have a direct connection by a road. So how can we test if two cities are directly connected and get the distance between the two cities? With a hash table using a tuple of two city coordinates as key it is easy:

For the procs insertDist() and checkDirectConnection() we use a trick to get the same results when we exchange the names: We sort the names alphabetically when we insert the distances and also when we query the distances. So we get the same result. Of course we could insert the tuple also twice instead, but as distances is the same in both direction sorting and inserting only ones makes some sense. Note that we used tuples for the coordinate pairs in the distances tables. Maybe the more obvious data type would be an array with two entries, as the array type is a container for homogeneous data, while a tuple can also contain different data types. But currently Nim supports tuples better in some situations, e.g. for automatic tuple unpacking. So often we use tuples when array would be the first choice maybe. array type would have the benefit of iteration over elements at runtime, while tuples have the benefit that we can access elements by names and by an integer constant. For performance array or tuple should make no difference here.

References:

sed -i -E 's/_([a-z])/\U\1/g' myfile.txt

nimble install regex

import regex

let r: Regex = re"\w\d"
let t1: string = "a1"
let t2 = "nim"
var m: RegexMatch
if match(t1, r, m):
  echo "match t1"

if match(t2, r, m):
  echo "match t2"

import regex

let r: Regex = re"A[a-z, A-Z]*(\d+)"
let t1: string = "Alex77"
let t2 = "nim"
var m: RegexMatch
if match(t1, r, m):
  echo "captured: ", m.group(0, t1)

if match(t2, r, m):
  echo "captured: ", m.group(0, t2)

Whenever we create regex patterns, we have to care for the fact if sub-matches should be greedy or not. In most cases greedy is the default, and we have to take some care when we need none-greedy behaviour. Greedy means just, that the regex engine captures as many characters as possible, while none-greedy capturing stops the capturing process early. Indeed these greedy/none-greedy capturing can be one of the most demanding tasks when we create larger and complicated patterns. Imagine that for our above example, we would have used the pattern re"A\w*(\d+)". For the same string "Alex77" we would then get the output @["7"]. The reason for that is, that \w* does a greedy processing, eating all but the last decimal digit, which it left to satisfy /d+. From the API documentation of the regex module we learn that we can specify \w*? instead to get a none-greedy processing, so both digits are left for \d+ and we get again @["77"] as output.

The use of escape sequences in regex patterns is another difficulty for beginners. The first problem can be that the Nim compiler may process the escape sequences already itself, while we intent to left them for the regex engine. We can avoid that when we use Nim’s raw strings, e.g. we can use triple quotes when we construct the pattern from individual strings as done in our next example. In an regex, we can use escape sequences to specify special literal characters, we may use \t for a literal tabulator for example. And finally, we may have to escape some punctuation characters like *, + or ? that have a special meaning for the regex engine when we intent to use that character as an ordinary literal. For example, to match a letter followed by a question mark, we have to use a pattern like "[a-z]\?" or "[a-z][?]". Inside of a square bracket we can use the punctuation characters without the need to escape them.

As next example, let us assume that we have to process a text file in which each lines starts with a name consisting of lower case letters, and three decimal numbers. The name and the three numbers can be separated by spaces, or by commas or semicolons:

To understand the pattern that we use in the above code, we have to know that we can use \s for a white-space character, so \s matches a single space or a tabular character. We could have used just a space literal instead, or the [ ] character class containing just a single space. And we have to know that we can use ? to specify an optional entity. We have split the total pattern into two parts, where the variable called h stands for the sequence of any number of white-space, followed optionally by a single comma or a single semicolon, followed again by any amount of white-space, that is finally followed by at least one decimal digit. As we want to capture the decimal numbers, the sequence of decimal digits is enclosed in round brackets. The total regex pattern is constructed by the subexpression [a-z] for at least one letter, followed three times by the integer pattern with the allowed separators. Note that we allow any amount of spaces or tabulators, but only a single comma or semicolon between the different entities. Note that the match() function of the regex module does always a full match, so a single space at the beginning or end of the text string would make the match fail. We could compensate for that by starting and ending the regex pattern with "\s*". Or we could use instead of match() the find() function, which search through the string looking for the first location where there is a match. When we use find(), we may use the special characters ^ and $ to match the start or end of the string, that is with find() and re"\s+$" we could find all strings which have trailing white-space. Note that find(text, re"^regex$", m) is the equivalent to the match() function.

The regex module provides us also with two replace() functions, which we can use to replace matched patterns with literal strings or captured and modified strings. The first replace() function uses as third argument a string, which is used for replacements and in which we can refer to captured groups with the symbols $N, where N is the index of the captured group starting at one. The second replace() function uses a function as third argument, that function gets an instance of the RegexMatch type as first parameter and returns the string replacement. We will use both variants of the replace() function to create a tiny app that we can use to fix typos in program and text files: Text files can contain typing errors, which includes two or more spaces between adjacent words, unneeded trailing white-space at the end of lines, and the use of a instead of an in front of words starting with a vocal. And program source code may use snake case for names instead of camelCase, e.g. line_counter instead of lineCounter. We will create a tool that can fix these four issues — ignoring the fact that an actual a/an replacement may corrupt program source code. To demonstrate the four issues, we have created this small test file — line three contains two unneeded spaces, and the last line has some unwanted trailing white space:

We will fix these four issues independent of each other, so we will try to find a regex that matches for each issue, and then use the replace() function to fix it.

We process our file with the issues line by line, using the lines() iterator to which we pass a file name and which gives us the individual lines of the file. We will start with the simplest task, that is removing trailing white-space. The search pattern for this issue is obviously "\s+$", that is at least one white-space at the line end, which we have to replace with an empty string. So we pass this regex pattern called trail and an empty string literal to the replace() function. Replacing a by an is also easy — we search for an a followed by white-space and a vocal, for which the regex pattern is the aan variable in above code. In this case we have to preserve the actual white-space and the vocal, so we enclose these in brackets to capture it. The replacing string is "an$1", where $1 stands for the captured white-space and the captured vocal. Replacing too much inter-word space is a bit more difficult. The actual issue is one white-space followed by one or more white-space, for which a possible match pattern is "\s\s+". But actually we do not want to remove all white-space consisting of more than one character, but only white-space between words. So multiple white-space at the beginning of a line should be preserved. One solution is, that we use the metacharacter \S, which matches all none-white-space characters, and then use this search pattern: "(\S\s)\s+(\S)". The pattern starts with a none white-space character, followed by a white-space, then at least one more whites-space character, and finally a none white-space character. We capture the two first characters, and the last one. This way we can replace the whole match with the two captures and we are done. Finally, we have to replace underscore characters followed by a lowercase letter with the capitalized letter. Some tools like sed provides the \U to capitalize a capture, but this is not available for the regex module. So we use the replace() variant which uses a proc as last parameter — to that proc the capture and the original string is passed, and that function should return the replacement string. The capture which we have to use to catch a snake element is obvious just "_([a-z])". We call the converter proc toUpper(), its parameters and its return type is specified by the regex API docs. But unfortunately the actual structure of the passed RegexMatch instance is not that detailed described. So we created some conditional echo() statements inside the body of our toUpper() proc to show us the structure of the parameters. When we compile our program with the -d:debugThis option, and run it, we get this output:

So the last string parameter is always the whole string that was passed as first argument to replace(), and m.group(0) is a sequence of slices for the first capture. We need only the first element of this seq, as we have only one capture, and we use that slice to extract the captured sub-string by use of s[m.group(0)[0]]. Finally we apply strutils.toUpperAscii() on this sub-string to capitalize it, and return that result.

When you run above program, you should get a text file with all issues fixed. You may redirect the output to a file with "\.t test.nim > newtest.nim" and load newtest.nim into an editor to proof that the trailing white-space is removed as well.

The use of regular expressions is not that easy, and makes in most cases not much sense in Nim. Maybe the largest problem of regular expressions is, that it is hard to understand patterns that we created some years ago, or that have been created by other people. And it is difficult to modify that patterns later. Maybe you should play a bit with regexes yourself now, and come back to this topic when you think that you need them. In this book we were only able to give a tiny introduction into the stuff — you will have to carefully study the API docs of the regex module and a lot of other resources in the Internet when you should intent to seriously use them. We should also mention, that while regexes are very powerful, for some tasks they work not that well, e.g. parsing math expressions with nested braces, or just skipping nested comments in some source code can be very difficult or even impossible.

References:

The comment on the top of the above example shows a partly sorted list of 7 integer numbers. The lowest three positions already contains the sorted numbers 1 to 3. The next four positions are still unsorted. For the sorting process we need the three indices i, j, k and the variable x to store an actual value for the comparison. The index j is the lower bound for the still unsorted range, j starts at zero obviously. The variable x stores the currently smallest value of the still unsorted range, and k is the index position of that value. Finally i is a counter that is used to step through all the values of the unsorted range. The outer loop is executed as long as j is smaller than the length of the sequence that we want to sort. We set i to the value of j, k to the same value and assume initially that s[i] is the smallest value from the still unsorted range. That value is stored in x. Then we execute the inner while loop until we have processed all elements of the still unsorted range. Whenever we find an element that is smaller than x, we store that position in k and the value in x. When the inner loop has finished, we exchange the smallest value with the first element of the still unsorted range. This way the sorted range increases, and the unsorted range decreases by one.

To test our sorting procedure we generate some random numbers, sort them and print the result. Selection sort is said to be of order O(n^2), with n being the number of values to sort. So the effort increases quadratic with the number of values. This is because we have to test all the still unsorted values just to increase the number of sorted values by one. Selection sort has a natural behavior, that is for an already sorted array the test s[i] < x would be always false and we would have to do no movement of values in that case. So performance is best for an already sorted or partly sorted list, and the sorting is stable in the sense that we do not move elements when it is not really necessary. In one of the following sections we will discuss the quicksort algorithm, which is not a stable sorting method: Elements with equal value may be move with quicksort. For plain numbers that does not really matter for the result, as numbers are indistinguishable. But when we sort objects, maybe persons by age, persons of same age would be exchanged by quicksort, which may not be desired.

Note that the code above is not really optimized for performance yet. One possible improvement may be to iterate the two loops not from zero to s.len, but in the opposite direction. In that way comparison of loop indices with a constant value, zero in this case, could be used to terminate the loop. Comparison with constants can be faster than comparison with actual variables, and comparison with zero is generally fastest. Note that we compared indices with s.len() in above code, which is not that bad, as len() is a field in the seq data structure, so the compiler should be smart and replace s.len with just a field access without proc call overhead.

Insertion sort is another simple sorting method, which some card players like to use: They hold two sets of cards in their hand, one unsorted set, and one sorted which is initially empty. They pick one card from the unsorted set and insert it at the right position in the already sorted set. That action is repeated until the unsorted set is empty. For our next example we sort our data not in place as we did it in the previous example, but we generate a new sorted copy:

The commented code in front of the above program code shows at the left the still unsorted numbers and at the right 3 already sorted numbers. For the sorting process we need two index variables j and k, and one variable to store the actual value that we have to insert called x. The variable k is used as the index of the top entry of the still unsorted range. To insert the value x in the already sorted result, we first reserve space for one more entry by calling setLen() and then iterate over the sorted values and move them one place to the top. We do that moving to the top as long as we have not already reached the bottom of the sorted range and as long as the current entry is larger than the value x which we want to insert. We take the values from the top of the unsorted range, as that is convenient, but of course we could pick an arbitrary element from the unsorted range.

Insertion sort has O(n^2) cost, as for each element that we take from the unsorted range we have to iterate over the sorted range to insert it. As we have to move elements before we can insert an element, insertion sort is slow for larger containers. Selection sort, which is also of O(n^2) should be faster, as it does not use an expensive shift of many elements.

When you look at the example code you may immediately find two possible improvements: We do not really need the variable x, as we can just use s[k] instead. The compiler should optimize the code so that the subscript operator is not executed multiple time for the same index k, so use of s[k] or x should make no difference for performance. And we call setLen() in the outer loop to increase the capacity of the result sequence by one each. Of course setting capacity only one time to the value of s would suffer, as obviously result has the same length as out input data. Another possible optimization would be to take advantage of the fact that the destination sequence is sorted, so that we would not have to do linear search to find the insertion position, but we could use a binary search. But that would be more complicated and the benefit would be not large.

As the name implies this sorting method is one of the fastest. We will explain it in some detail with various variants as it can teach us two important concepts: Recursion and avoiding recursion by use of a stack container.

The idea of the QuickSort algorithm is simple, and the code is also simple and short, but we have to care for some details like exact index stop positions. Generally sorting seems to be a O(n^2) operation, at least from the two traditional sorting methods, Insertion- and Selection-Sort it seems to be the case. So doubling the container size seems to increase the needed sorting time by a factor of four, which is really bad for large arrays or sequences. The trick of QuickSort is, that instead of sorting a container with n elements, we just sort the first half and the second half separately, each with approx n / 2 entries. This would be faster, as 2 * (n/2)^2 is only half of (n)^2. And we do apply this trick in a recursive manner on each halved range, until the range is reduced to only one or two entries. But to make this work we have to partition the full range in the first half r1 so that all entries of range r1 are smaller or equal to a median value x, and so that all entries in the second half r2 are all greater or equal to the median x. Let us consider an example with six numbers:

To partition that set of numbers we have only to exchange the numbers 5 and 1, with the value 5 or 4 being a possible median. Exchanging numbers in an array or a seq is a fast O(n) operation, we have to iterate the container only once. The problem is finding the median. For picking a perfect median we would need a sorted container so that we could pick the center entry. But of course our container is unsorted, if it would be sorted our work would be done already. Note that even summing up all entries and dividing by n would give only the average value, not the median. Average and median can be very different, e.g. for many small numbers and a few very large ones. But in practice picking an estimation for the median, maybe picking one by random from the whole range or picking the center entry is good enough. That choice will not really half the whole range in each step, but on average it splits the range in two parts with not too different size. This works really well when the input data looks like random numbers, but it may work bad in some unlikely cases when all the input numbers are equal or are already sorted. Already sorted can indeed occur — for that case picking the center elements of the range gives the perfect median, so we will choose that strategy.

Partitioning the full range is basically very simple: We move from the left to right with index i and stop when we find a value s[i] that is not smaller than our median x. And we do the same from the right to the left with index j until we find a value that is not greater than median x. After we have done that, we can just exchange the values at s[i] and s[j] and continue. We continue until i is close to j. The difficult part is to handle this terminating condition exactly, that is to stop exactly at the right position so that the first half really contains all entries with value less or equal to median x, and that the second half contains only entries with value equal or greater than median x. To make it more clear: Such a partition decouples the two ranges. Sorting the whole range would result in the same state as sorting first and second range on its own.

For writing our actual sorting function we use the fact that Nim like most modern programming languages support recursion, that is a function can call itself again. We saw at the beginning of the book a few examples for that. So we can pass our function the initial full container, then the function can partition the container in first and second half and call itself again on the two parts. This way the actual size of the ranges to sort decreases, and finally recursion stops when the range contains only one or two elements. For size one we have to do nothing at all, but for size two we may have to exchange the two elements if order is wrong.

A naive implementation may create for each function call two new sequences for the two parts, partition the initial sequence by inserting the values in one of the new shorter sequences, call itself on both parts and finally join the parts and return it — as result or as var parameter. But that would be really slow. A much simpler and faster solution is, when we work all the time on the same container, and just tell the function which range the function has to work on. So we pass to the function the whole sequence and two integers a and b which tell it the range to process: s[a], s[a +1] .. s[b].

The function qsort() does the whole work. It is called from function quicksort() passing it the whole sequence and the interval to sort. For the first call the interval is the whole content of the seq, from s.low to s.high. Function qsort() first asserts that the range is valid, that is that b > a and that both indices are valid positions in the sequence s. That check makes it easier to find stupid errors, the assert is automatically removed when we compile finally with -d:release. We set the iterating indices i and j to the interval boundaries a and b and enter an outer loop. In that outer loop we let run i and j to the center of the interval as long as the actual entry at the position i or j belongs in the range. If both inner loops have stopped, we swap the entries at position i and j. As positions i and j contain now again valid entries, we can move both indices one step further to the center. If i and j becomes the same we are done. Unfortunately both may stop too late, so we move both one position back after the outer loop has terminated. You may create a small example with pen and paper to recognize how the indices behave in detail and why fixing by one is necessary. The remainder of the qsort() proc is really easy: For both partitions we check if the interval size is still larger than two entries, in that case we call qsort() again to continue with partition and sorting. But if the size is two entries, then we just swap() them if the order is wrong. If the size of the range is just one, we have nothing to do at all.

We test our quicksort() proc by calling it from another proc called main(). In that main() we fill a seq with random integer values, and shuffle() and sort() it a few times. Shuffle() reorders the entries by random. After our call of quicksort() we call isSorted() from Nim’s standard library to check the success of our sorting. After these tests, which does some warmup of the CPU for us, we add more random entries and again sort and test it, while we record the needed time with module monotimes as we did before in the Timers section. To get a feeling about the performance of our sorting proc we shuffle() again and sort this time with the sort() proc from Nim’s algorithm module. Sort() from algorithm module uses currently another sorting method called merge sort, which has the advantage that it is a stable sorting algorithm, but it may be a bit slower than quicksort. And sort() from algorithm may pass a cmp() proc around, which may cost some performance, while our plain, non generic proc compares entries directly with < and > operators. So it is not surprising that our proc is a bit faster.

You may wonder if it is really necessary to pass the sequence s for each call of qsort(), as for all time the same seq is used. Indeed Nim support nested procs, so we could just make qsort() proc local to quicksort() proc and let qsort() work (as a closure) on the s variable of proc quicksort(). But this does not compile currently, sequences can not be used by closure procs. But actually passing the sequence to proc qsort() should be only a minimal overhead.

One general problem of QuickSort is that the sort is not stable. When we sort an already sorted sequence again, entries with same value may move. For plain numbers that is not really a problem, we do not really notice it, as we can’t mark a number in some way, plain numbers are indistinguishable just as elementary particles like electrons and protons are. But when we sort a container with objects by some field, then we notice that objects with same value for sorting may move. The other problem of QuickSort is a generally problem of recursive algorithm: Each new call of a proc generates some stack usage, as proc parameters may be passed on the stack and because the proc may allocate its local data variables on the stack. So many nested calls may need a very large stack, the program may fail with a stack overflow error. Generally, we have no real problem with stack overflow, as for each partition the size of the two new partition is nearly halved, so that process stops soon. But imagine someone prepares a special data set for our sort proc. That data may be prepared in such a fashion that at the center of each range, where we pick the estimated median value from, always an extreme values is stored. So our partition would work very badly, in each step we would get a new range with only one element, and one with n - 1 elements. So the recursion dept would go very deep, and the performance would be very bad also. Preparing such a data set would be difficult, but possible in theory. One way to protect us from that attack would be to select the median by random. But unfortunately all strategies different from picking the leftmost, the center, or the rightmost entry as median are not very fast and make the whole sorting significantly slower. Note that the strategy of not picking a single element as median, but calculating a median value, works generally, but has some shortcomings: s[a] div 2 + s[b] div 2 would not work when both values are odd, as we then would get a value that can be smaller than all of our entries and our function would fail. We would have to add one to the average value when both summands are odd, and that fix cost performance again. And calculating the average by (s[a] + s[b]) div 2 could generate an overflow when both summands are large.

Because of the stack size restrictions we have a good motivation to show how we can replace recursion with plain iteration, when we provide a "buffer" variable that acts as a data stack. For each new partition of our data we have to put only the two bounds a and b on that data stack, which is not that much as a recursive proc would put on the real computer stack. The modifications to our code from above are tiny:[42]

We added a variable called stack which is a seq which stores integer tuples. The qsort() proc first stores the borders of the whole seq on the stack, and then executes a loop which takes in each iteration a set of two borders from the stack and processes that range. It may sound a bit strange that we start by putting the whole range onto the stack and then took it from stack immediately at the start of the loop. But that makes more sense when we look at the bottom of the qsort() proc. Instead of recursively calling qsort() again, we just put the borders of the two new partitions on the stack and continue. The whole process terminates when the stack becomes empty, as then all partitions are processed. Note that the actual partition code and the main() proc are still unchanged. We added a maxStackLen variable to get a feeling how large our stack has to be. Actually not that large, as the partition size shrinks in a logarithmic way. So we could replace the seq that we use now as stack with a plain array, as sequences have same overhead and the add() is slower than plain index access. But how can we prepare for worst case attacks? Indeed there exists a simple solution: Worst case occurs, when first we put a tiny one element range on the stack and then the large one, as we would continue with the large one in the same way in the next loop iteration. The other way round would be fine. If we put the tiny range on the stack last, next iteration would pick that one and iteration would stop immediately or at least very soon, as ranges drops to two or one entries. When an iteration for a range stops, all ranges pushed to the stack are removed already again, so total stack size will never become large. So the trick is to just sort the partitions in a way that we put the larger partition first on the stack, and the smaller partition second. So next iteration picks the smaller one and the whole process stops soon. This way a stack array of 64 entries should be enough, as max needed stack size should be log2(2^64) to sort a seq with 2^64 entries.

Instead of processing the two new partitions at the end of the qsort() proc each, we apply only one processing code block now, which we execute in a loop that is executed two times. At the start of that loop we assign the actual interval boundaries to the variables c and d. That assignment depends on the actual loop index u, so that we push the larger range always first on the stack. You may modify the condition u == 0 to u != 0 and observe what happens to the maximum used stack dept. We could write that condition also with a boolean loop variable and a xor operator like

We should not believe all what we think

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: array[64, (int, int)]
  var stackPtr: int = -1 # empty
  var maxStackLen: int
  var a = s.low
  var b = s.high
  while true:
    if b - a == 1: # done with actual interval, but we may have to swap()
      if s[a] > s[b]:
        swap(s[a], s[b])
    if b - a > 1: # interval has still more than two entries, so continue
      discard
    elif stackPtr >= 0: # get next interval from stack
      (a, b) = stack[stackPtr]; dec(stackPtr)
    else:
      break # all done
    if stackPtr > maxStackLen:
      maxStackLen = stackPtr
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    # let x = s[a .. b].max # worst case test! Slow, test with smaller container size.
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i); dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    if (i - a < b - j): # put large interval on stack and cont. directly with the small
      swap(i, b)
      swap(a, j)
    if i - a > 1: # interval has more than two entries, needs further processing
      inc(stackPtr) # inc before push!
      stack[stackPtr] = (a, i)
    elif i - a > 0: # two entries, we may have to swap()
      if s[a] > s[i]:
        swap(s[a], s[i])
    (a, b) = (j, b) # the smaller interval, we continue with that one
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

    while true:
      if s[i] < x:
        inc(i)
      elif s[j] > x:
        dec(j)
      else:
        if i < j:
          swap(s[i], s[j])
          inc(i); dec(j)
        else:
          break
    dec(i); inc(j)

Initially we did not intend to discuss the actual MergeSort algorithm at all, as it is a bit more complicated and whenever we may have seen a sketch of it somewhere, it is generally not easy to remember details.[43] But MergeSort is indeed an important algorithm, it is used by default in Nim’s standard library and as we have discussed QuickSort already in some detail, we should be prepared for MergeSort now. When we regard the name Merge, which is some form of joining multiple sources to one destination, we may begin to remember the idea of MergeSort: The trick of QuickSort was, that we tried to split in a recursive manner the set of all container elements into two subsets, which we can process separately. That improves performance, as sorting is basically an O(n^2) process, and 2 * (n/2)^2 is only half of n^2. For QuickSort we partitioned the initial range into two ranges a and b, where all elements of range a are less or equal to a median element x, and all elements of range b are greater or equal to the median x. That way we decoupled the two ranges, we can sort a and b independently and get a fully sorted range. MergeSort starts also with splitting the full range into two parts, but it really only splits, without any form of rearrangement. Then it continues with sorting each part independently. That sounds strange at first, as we get two sorted parts a and b, but of course we can not simple append one to the other. The idea of the whole algorithm become clear immediately when we think about how we can find the smallest elements of the joined content from a and b. That one is obviously the smallest value of a, or the smallest value of b, min(a, b) = min(min(a), min(b)). But when a and b are already sorted, then the minimal value of each is the first element, and so one of these elements at index position zero is the smallest one for a and b joined. And this condition holds even when we pick and remove the smallest element from a or b.

So the basic algorithm is this: Split the whole container in parts a and b, sort them separately. Then create a new, sorted container by iterative picking the first elements from a or b, which ever is actually the smaller one.

Unfortunately it seems to be impossible to sort the initially container in place in this way, as we would take always elements from the front from both and put them at the front of the destination, so that newly inserted elements could overwrite still unprocessed elements. So we will try to give a sketch of an very slow unoptimized algorithm which creates and returns a new sorted container first to lean the fundamental idea of the algorithm:

The example above is indeed not very complicated, the most daunting task is to get all the indices right. Due to the involved recursion and the fact that we have to do the sorting of the two parts first, before we can join them, debugging would be not easy. So we added many asserts to early find stupid errors.

The function mSort() starts by checking if the range to sort has only one or two entries and handles this simple case directly. Then we allocate the result sequence, find the center position m of the interval a and b and sort the intervals a .. m and m + 1 .. b each into a new sequence s1 and s2. We have decided that we will do the merging of s1 and s2 into the result sequence from the back, starting with the largest elements. This way we can count down to zero, which is a bit faster and simpler. We do an actual merging as long as s1 and s2 have still elements left. An we do the merge from end to start, we have to always pick the largest element from s1 or s2. If we have used all the elements from at least one of the sequences s1 or s2, then there is nothing more to merge, we can just copy the remaining elements from the other seq that has elements left. Of course the above example is very slow, as we allocate the result and additional the sequences s1 and s2, and as we have to copy many elements.

When we now think again about the problem, we get the feeling that allocating three sequences is really too much. When we regard both, in place sorting and sorting with a return value, we may discover that in place sorting allows us to partly reuse the passed container, and we have only to allocate one additional seq with half the size of the passed container. We copy the second half of the passed container into a newly allocated seq, and then can merge values from the new seq and from the first half of the passed seq to positions starting at the end of the passed seq, without overwriting values that we still have to process.

This way our code becomes even shorter and basically simpler — we have only to care very exactly to use the right index positions.

For the merging process, we first test if one of the source areas is already exhausted. If all entries from the newly allocated seq sh are consumed, then we can just stop, as continuing would only copy elements of the passed container with equal index positions. And if all elements from the first half of the passed container are consumed, we can just copy the elements from sh into the passed var container. Only if both sources have elements to process left, we have to do the actual merging.

When we run the program above, we may find that it is about 50% slower than our QuickSort functions. The reason for that may be that we have to allocate the temporary seq sh, fill it with values, and merge it back. And the total memory consumption is high, our recursive function calls consumes for the buffers sh totally the same amount as the initial container. The advantages of MergeSort is that there is no worst case as for QuickSort, as we have not to select a median but can split the range just at the center, and that the sort is stable, that is merging does not exchange the position of elements with same value.footnote[ Well to ensure this, we may have to check the actual merge condition, do we have to test for < or <=. Currently we do not really care for that, but is is clear and well known that merge sort can be stable.]

When we think a bit more about the algorithm above and maybe try to sketch the recursive steps for a short sequence with pencil and paper we get the strong feeling that the additional buffer sh is only needed for the merging step, and as the merging process occurs from bottom to top (containers with only one or two entries are returned immediately, which are merged to a larger section, and these larger section is again merged …​) one single buffer could be used. So we have modified our example again. Now the quicksort() proc allocates the buffer seq with half the size of the actual data container, and we pass that buffer recursively to the qsort() proc and use it for the merging only. We call recursively qsort() on the first and second half of the full range that we have to sort, then copy the sorted second half into the buffer and merge the fist half and the buffer into the final location.

An additional tiny performance improvement results from the fact that we now pass the seq s and the buffer sh as openArrays. This is generally a good idea, as we can so sort arrays also with the same sorting proc, and it improves performance, as this way the actual data buffer is directly passed to the qsort() proc, while passing a seq means that we pass the opaque seq structure which contains a pointer to the actual data. In the example above we do not only call isSorted() to prove our result, but we really sort a copy of our data with a sorting routine from Nim’s standard lib to ensure that our result is not only sorted data, but that it is indeed based on the actual values. That is a good idea, because although the algorithm is simple, getting some indices wrong may give us wrong results.

Our recursive merge sort routine is really not that bad. It does a fast stable sort and needs only a single buffer of half the size of our actual data. As the interval size is halved in each recursion step, the max recursion depth should be only 64 for a gigantic container with 2^64 elements. As the recursion occurs in a dept first fashion, that is msort() calls itself until range size is only one or two elements, then recursion continues in other branches of the whole sorting three, there should be never more than log2(n) actual recursion steps stored on the CPU stack. Non recursive, iterative merge sort algorithm exists, but converting the recursive algorithm in an iterative one is not that simple as for the QuickSort case. The reason is that qsort() has first to partition the input range, then call msort() on both subranges and finally do the merging. We will not try to present in this book an iterative msort(), which you may find in textbooks or somewhere in the internet, as that would be a bit too much for an introducing course.

We did the QuickSort and the MergeSort in a top down fashion, that is we split the initial container in two subpartitions and continue in this way until we have only ranges with one or two entries, and than do the merging from bottom to top. For MergeSort we could just start from the bottom joining single adjacent elements to sorted tuples of two entries, when done with that merging the tuples of two to sorted tuples of 4 and so on. This would work really well when the initial number of elements in our container is a power of two, and it would work well iterative without recursion. Unfortunately in most cases the container size is not a power of two, so such a bottom up merge sort needs some math to get all the ranges sizes right. But the bottom up process has a big disadvantage on modern hardware, as it has no locality for element access operations: We would iterate repeatedly over all the container entries in sequential order, so the CPU cache can not support our element access operation that well.

Other known sorting algorithms are the easy, funny and slow BubbleSort, or Shell- and Shaker-Sort. But these are not used in practice. As an exercise you can try to make our QuickSort or MergeSort generic and pass a cmp() proc, and make it work for sorting in ascending and descending order. Or, you may try to fall back to selection sort when the partitions become small. In theory SelectionSort is faster for ranges of only a few dozen elements, but when we have to do a decision which one to use inside of the qsort() or msort() proc, then this decision compensates generally the advantages again, so that the net benefit is tiny. Of course we would have to test all of our sorting procs for special cases, that is for seqs of length 0, 1 or two, and for sequences with all entries equal, all inversely sorted or presorted. And we would have to check how performance is when we sort not containers containing plain data like numbers, but containers which elements are objects, strings or again arrays or sequences. string sorting is special for various reasons: strings in Nim are an opaque object with a pointer to the actual data. This is some indirection, and the actual data can be located somewhere in the RAM in a cache unfriendly manner, so the actual comparison process can be slow. Swapping of strings is also special, as swap() generally just does a pointer exchange for the data areas, and does not have to copy the actual data. For sorting containers where each entry is an array (of characters), swap would have to copy the data content.

Finally we should mention that the Python language uses a complicated sorting algorithm called TimSort which is a smart mix of various sorting algorithm.

References:

• https://en.wikipedia.org/wiki/Timsort

Removing duplicates from containers is a common programming task. When we know in advance, that we do not want to store the same data value more than once, and that the order of the stored values does not matter, then we may consider using some form of sets or hash sets. If insertion order matters, then the standard library may provide some form of ordered or sorted sets for us. But for this exercise we will assume that we have values stored in a sequence, and we want to remove the adjacent duplicates. A typical use case for that is when we have stored a path of 2D or 3D positions. When we insert, move or delete a position value, it may occur that we get duplicates, with zero spatial distance between the two neighbored positions. In that case it is generally desired to remove the duplicate. A similar use case may be a text file stored as a sequence of words, where adjacent duplicated words may indicate a typo. To keep our example short and simple, we will use as data type a seq[int]. Using other data types or creating a generic proc should be not difficult for the reader.

Before you continue looking at our provided example code, you may think about this task yourself, maybe take a piece of paper and a pencil and sketch the algorithm. It is really not difficult, maybe too easy for you when you have carefully studied the proceeding sections of the book or when you have already some programming experience. When we create a proc for this task we have to decide first if the proc should work on the passed in seq in-place or if it should return the processed result and leave the original data unchanged. Often returning a copy is easier, but for our task the common use case seems to be more an algorithm that works in place. So we will provide the in place algorithm here and leave the version returning a processed result as an optional exercise to the reader. Note that returning a processed copy needs to allocate the seq for the result, and maybe later the memory management system has to free the result data again, which is some additional effort. So we may guess that the in-place proc is faster, as long as we do not really need the copy. When needed we can use the dup() macro of the sugar module to use our in place proc as one that works on a copy and returns this copy, without modifying the input.

The basic idea of our algorithm is that we iterate through the whole seq and pick an element at the current location only if it is not identical to the proceeding element. For the case that our seq is empty or contains only one single element, we have obviously noting to do and can return immediately. So our code may look like

We use two positions, the actual position in the input data denoted as i, and the destination position d. Both start with the default value zero. To keep full control over the iterating process, we do not use a for iterator in this case, but a plain while loop for the index of the actual position. In the while loop body we compare the value at the current index position s[i] with the value that we picked before, which is s[d]. If the values are not identical, we pick the current value s[i], otherwise we just skip it. With picking a value we mean that we copy it from index position i to index position d. Of course this can only work, when d is never larger than i, as otherwise we would destroy our still unprocessed input data at positions s[i + 1]. The loop body is really simple, but getting the indices right needs some care: We have to ensure that i and d starts at the right positions, that we increase i and d when necessary, and that the loop terminates when all input data is processed. Obviously the first comparison should compare s[d == 0] with s[i == 1]. So d and i can get initial values zero each, when we increase i already each time at the start of the loop. The destination position d starts with zero, as we always accept the first element, and d increases only when we have accepted one more element. The loop is executed as long as i is less than s.high. Finally we have to set the new length of s to d + 1. The value d + 1 results from the fact that we always accept the first element, and for each more accepted element d is increased, so the total number of accepted elements is d + 1. The carefully reader may wonder if the first two lines of the proc, where we test for the trivial case, are really necessary, or if these cases can be covered by our while loop already. Well, generally it is a good idea to avoid unnecessary tests for trivial cases when possible, as that tests may increase code size and cost some tiny bit of performance. But in this case we have two trivial cases — empty seq and seq with only one element, which can be not covered well with a loop with only one simple termination condition. And of course we should try to make the condition of the while loop as simple as possible, that is avoid additional boolean conditions with and or or operators for best performance. Further you may wonder if our picking strategy is really optimal, as for a seq with no adjacent duplicates we still copy all the elements. Yes indeed, but for the general case with duplicates we have to do the copy, and such a copy operation is really fast. And additional tests with an if condition should cost some performance. Maybe we could have used two loops in the proc body, one that just accepts elements without a copy operation as long as no duplicates are found, and a second loop like the one from above which then has to do copy operations to move the elements to the front. You may try that yourself and measure the performance for various input data.

The difference of two arrays or sequences A and B is the set (A - B) of values that are contained in A but not in B.

Actually such difference of array or seq containers is not needed that often, that is why that function may not be provided by the Nim standard library.[44] Building such kind of differences is much easier and faster with sets or hash sets, so whenever possible we should use these containers from the beginning when we know in advance that we have to build differences. But sometimes we just have arrays or sequences, and then we may notice that we need the difference. When the order of the elements does not matter, we may just convert both containers to sets or hash sets, build the difference and then maybe convert that difference back to a seq. But there are use cases where we really want to work with array or seq containers, maybe because we want to iterate over the container, want that values can be contained multiple times or always keep the insertion order.

So let’s create an algorithm to do this task. A naive strategy would be to iterate over container A and delete each element that is also contained in B. But that would be slow, and may not work at all, as deleting elements while we iterate over the seq does generally not work at all. We will create a proc called `-` which can be used as an operator to build the difference of two arrays or sequences and which returns the difference as a new seq, and a `-=` proc which removes the elements of b from a in place and is also used as an operator.

To make the lookup for elements contained in b fast, we convert b to a hash set, for which lookup time is in principle independent of the size of the container, which is called O(1) in the big O notation. We make the two procs generic and use the data type open array for the two passed arguments so that our procs can be used for arrays as well as for sequences. The exception is the first var parameter of the `-=` proc, which has to be a seq obviously, as arrays have a fixed size and can not shrink. For the `-` proc we pre-allocate the returned result variable with a size of a.len, so that we can avoid re-allocations. Then we iterate over a with a for loop, and copy the current element to the result seq when the value in not contained in the hash set. We use the subscript operator [] to copy the picked elements at position i in the result seq, which is faster than starting with an empty result seq and appending the picked elements. As we initialize the result seq with the size of container a, we have finally to call result.setLen(i) to shrink the size to the number of actually picked elements. The presented `-=` proc is a bit more complicated, as we process seq a in place. We use an approach similar as we did in the deTwin() proc in the previous section, that is we use two index positions i and j, and copy elements from current position i to position j if the value is not contained in the lookup set. Again finally we have to set the size of seq a to the number of picked elements.

While the two presented procs may be actually useful in same cases, they are more presented as an exercise here. As a smart user of the Nim forum showed us, we can get a very similar behaviour by use of the filter() proc in combination with the => operator of the sugar module:

References:

• https://forum.nim-lang.org/t/7753

Maybe you can remember that some decades ago your parents have used phone books and dictionaries build of paper sheets, filled with printed text sorted alphabetically? Well, that alphabetically ordering was done with a purpose: When searching for a name or a word, we could just open the book somewhere in the middle. If the name or word that we searched for was ordered alphabetically before the content of the current page, then we continued searching for that term in the lower half, otherwise in the upper half. That procedure was continued until the searched entry was found, or until it was observed that it was not available at all. This type of search in an ordered data set is called binary search, half-interval search, logarithmic search or binary chop. As each repetition halves the remaining data set, it is much faster than a linear search in unordered data.[45]

To use this type of search strategy on the computer, we store our data sorted in an array or a seq. Creating a proc to do the search is basically very easy, but we have to care for some details:

To keep our example code as simple as possible, we do our search on an ordered array or seq of integers. The value which we search for is passed as second integer argument to the proc called binarySearch(). The first tree lines of the example program shows some example data consisting of the ordered numbers 1 .. 8. Here we use a consecutive sequence of numbers, but the actual numbers are fully arbitrary, as long as the sequence is ordered by the value of the numbers in ascending order. Let a be the index of the lowest number in the seq, and b be the index of the largest number. If the number we search for is contained in the seq, than that number must be located at an index position greater or equal to a and an index position less or equal to index position b. For the index position p near the center the obvious choice is (a + b) mod 2. For our example, we assume that we search for a value of 7. The first index position of p is (0 + 7) div 2, which is 3 containing value 4, which is lower than the searched value 7. So we would have to continue our search in the upper half, setting the new lower bound of the range to search to p or p+1. The upper boundary remains unchanged for this case, and we continue with a new value p = (a + b) mod 2.

The program code follows this strategy in a straight way. We start with a == s.low and b == s.high, and the loop continues as long as a <= b. Note that we use as new boundaries not the value of p, but p + 1 if we continue with the upper half, and p - 1 when we continue with the lower half of our data. We can do that offset of one, as we have investigated position p already. This offset of one does not only speed up things, as the new interval is smaller by one this way, but actually guarantees that the interval size permanently shrinks and the algorithm terminates always. Without that offset, for the case b == (a+1) we would get a value p == (a + a + 1) mod 2, which is again a, and we might set the new a then to p, which is again the previous a. So the range would not always shrink, and our algorithm would not really terminate for some data values. You may test that yourself when you remove the offset — the algorithm would not terminate for some data. And finally you should try to make that proc generic, and then maybe search for a word in an ordered list of words.

References

• https://en.wikipedia.org/wiki/Binary_search_algorithm

Have you ever asked yourself what actually is happening when we print the value of an integer variable on the screen, maybe by use of the echo() procedure? Before echo() can print the value, the integer has to be converted to a text string somehow. Some people may think that this conversion is trivial. But as we know already that in our computer all the data is stored in abstract binary form, we know better. But maybe there is some magic available to do this task? Well when we regard the existence of C libs or the Nim standard library as that magic solution, then the answer is yes. But in this section we will assume that we will not use a C library or a function of the Nim standard library for the conversion of integers to strings, but do it our self. Indeed this conversion task is an interesting exercise, from which we can learn a lot, much more than from using a gaming lib and moving some sprites over the screen. Even when you already know how to do it, you may learn some new. We will start with the question, how we can convert an int i with a numeric value 0 <= i <= 9 to a single character digit matching this value, and then we will present a first procedure to convert larger integers to strings. After that we will try to improve that first procedure, we will make it generic and will investigate which problems may occur on restricted hardware like small microcontrollers and embedded systems.

As there is no magic available, let us recall how we can print the characters 0 .. 9: Well, first we have to remember that the 256 ASCII characters maps directly to the integers 0 .. 255, e.g. the character A is mapped to the integer value 65. We can use the conversion functions int() or ord(), and char() to convert between the two data types, where these conversion functions does no real work at all, the content of the variable is just interpreted as a different type. That is, a plain cast would do the same for us:

The same conversions work of course for all the 256 ASCII characters, which includes the decimal digits 0 .. 9. So one way to print the 10 digits is

This works,because the 10 decimal digits follow each other in the ASCII table. As we can not remember the position of the digit '0' in that table, we get the position by int('0'). Note that int('0') and ord('0') is basically the same here, we have no real preference and use both alternately. The ord() function is generic, always returns an int and works for ordinal types, enums with holes and distinct ordinal types, while the int() functions have the advantage that we have them for different sizes like int8() and also for unsigned results. We strongly hope that you know well the difference between the int value 0 and the decimal character digit '0' — if not you may read again the section about characters in part II of the book, see Characters.

With these introduction, you may already have an idea how we can get the decimal digit for the lowest decimal place of an arbitrary integer value v: char(v mod 10 + ord('0')). This works, because v mod 10 is the numeric value of the lowest decimal place, that is a value between 0 and 9, and when we add ord('0') we get the corresponding position in the ASCII table. Finally we use char() to convert that numeric value to a char data type, which is only a plain cast, the compiler reinterprets the bit pattern as a character. So we are mostly done. To get the following digits, we just divide the initial integer value by ten to move all one position to the right, and then we continue with the initial step. We repeat that until the division by ten gives zero, then we are done. That division by ten may be still confusing for you — we know that in decimal notation a division by ten is a shift right, but why does that work for a number which is stored in binary form in the computer memory? Indeed it is a bit confusing. The division by ten works, because it is a pure mathematical, abstract division operation, fully independent from the actual representation of the number. Imagine you have a number in the range 10 .. 19. Now divide that number by ten. Independent how the number is stored, we will get a new number in the range 0 .. 9, and that value we can convert to a digit with the method shown above.

So following this strategy, we may get a first intToStr() procedure that may look like this one:

As output we would get

Not that bad, but unfortunately we get the digits in reversed order. And for negative numbers it would not work yet. But that can be easily fixed. What above code does should be obvious from the discussion before: We copy the passed integer argument a into a local variable v of same data type, so that we can modify it, and in the while loop body we extract the lowest digit and then divide the value by ten to shift it down to the right. We have to use a while true: loop with a break statement, because we need at least one loop execution to get at least one digit, but Nim does not support repeat loops as known from languages like Pascal. In the loop body we apply the discussed operation to get the digit of the lowest place, then divide the actual value by ten and continue, as long that value is not already zero. When it is zero we can leave the loop, as we do not intend to print leading zeros.

Creating a proc that prints the digits in the correct order and that can print the minus sign for negative values is straight forward:

We use an array of character for temporary storing the decimal places, and finally copy the digits into the result string. We pre-allocate the string with the correct size, and use the subscript operator [] instead of add() to insert the digits for performance reasons. Initially we create a copy v with positive sign of the passed integer argument a, and when the argument was initially negative, then we add an additional minus sign to the temporary array, which is finally also copied to the result string. All this is not difficult, we have only to care that we get all the indices right. A tiny problem is, that when the passed integer argument has the value low(int), then applying abs() would generate an overflow error, see section Binary Numbers in part II of the book if you forgot it. We fix for that by returning just the correct string for that unique negative value for now.

The above proc looks not that bad, but maybe we can improve it, maybe we can avoid the temporary array? The actual problem is, that we do not know how many total digits the integer argument will need in advance, and so it is impossible to position all the digits at the correct position in the result string. A possible solution is to use a function that gives us the the number of decimal places of an integer number. Indeed we have such a function available, it is math.log10(). Remember, log10(1) is zero, log10(10) is one, log10(100) is two and so on. So basically what we need. The logarithm function is not that slow on modern desktop computers, so it should be OK to use it. At the end of this section we will consider how we may replace it for tiny microcontrollers which do not provide a FPU. The improved intToStr() proc may look like this one:

This proc is very similar to the one before. We call log10() to get a measure for the number of needed digits. Remember that the logarithm is undefined for the argument value zero, for that value we use the default value i == 0. Actually in all cases i + 1 is the total number of digits that we have to generate — for the case that we have to generate a minus sign we increase i by one. We pre-allocate a result string with i + 1 positions, and put a minus sign at position zero, which is overwritten in the while loop when the argument was not negative. As we know the the total number of digits of our number we can use the variable i to put the digits at the correct positions in the while loop. The careful reader may wonder if log10(v.float).int will really work for sure for all integer arguments of v, or if we better should round the argument like log10(v.float + 0.5).int. Indeed, with that rounding we should be save.

The next task is to avoid the initial test for int.low. We really should remove that special case when we prepare to make the proc generic later. A possible solution is, that we work with uint64 instead with int in the proc body, as in

When we add 1 to int.low, then we can invert the sign, and convert the value to unit64. To the uint64 value we have to add again 1 to get the initial sequence of digits. And now we can make the proc generic:

We use as parameter type SomeInteger, which allows signed and unsigned ints of all byte sizes, and in the proc we test with is SomeSignedInt: if we have to care for the sign and in case of value int.low for overflow. The advantage of this proc is, that it works for all integer types, signed and unsigned. But one disadvantage is, that always the data type uint64 is used, which may be not available on microcontroller CPUs. Let us see how a proc for only unsigned types may look:

That one is really simple and short, so maybe it would indeed make sense to use this one for the unsigned types. And we do not need the unit64 type, so on a system with no native 8 byte integers that proc should work fine.

All the previous examples have used log10() to determine the number of digits for the passed argument value. On microcontrollers log10() may be not available at all, or may be very slow. So let us investigate at the end of this section how we can replace it. The basic idea is, that we repeatedly divide the argument by ten, until we get the result zero, counting the number of needed divisions. An equally approach is to start with a variable with start value one and multiply with ten, until the result is larger than our function argument. As division is generally slower than multiplication, and on microcontrollers a native div operation may be not available at all, we will try to use multiply operations. So we may start with a proc like

Can you see the problem? What will happen when we pass int.high as argument?

So a working proc is this:

We do the math with an uint64 type in the proc. For the case that the argument is an 8 byte type, we may get an overflow in the while loop, which we prevent by doing one division before the loop already. Actually for improved performance instead of a division by ten we may do a division by a larger power of ten, and fix the start value for result accordingly. One disadvantage of that generic proc is again, that an uint64 type is used for the math, which is fine on a desktop PC, but may work bad on restricted hardware. So this variant seems to be a better solution:

Here we do a single div operation if the argument is larger than 9, but do all the math with the same type as the argument type. The div operation may be still slow on a microcontroller, but our intToStr() proc has also used div operations. Indeed doing intToStr() conversions on a tiny 8 bit microcontroller is not really a good idea.

For determining the number of digits of integer numbers, you will find many more solution in the internet. Sometimes this is called log10() for integer numbers. Some functions try to use the logarithm with base 2, which is related to finding the highest set bit of a number, some other functions use tabular data or a sequence of if or case statements. As the performance of that functions depends on the actual hardware, there exists not really the best solution for all cases.

For the intToStr() function you should also find very good solutions in Nim’s standard library. Note that it was not our goal in this section to present a really good solution, the idea was more to show you how such a tasks can be solved in principle, and how we can improve or modify solutions, and how we can use Nim’s generics to get one function for multiple data types. Note that the presented procs are only minimal tested, and are tested only on a 64 bit desktop OS. So they may not work on systems where Nim’s int type is 32 bit, or for microcontrollers and embedded systems. But you have learned enough now so you could fix it for that cases.

As possible exercises for the reader we may suggest to create a similar proc called strToInt() that converts a numeric string to an integer number, or to convert between strings and float numbers. The first one is easy, you would build the int value by continuously multiplying the digit value with its correct power of ten matching its position in the string. The float conversion is more difficult, in one weekend you may get some working code, but perfect solutions like the ryu or dragonbox algorithm are very complicated.

No, not yet. We know that for many people game programming is the initial motivation to start with computer programming at all, and so a larger section about this topic would make indeed some sense. But there are some reasons why we do not have this section currently. Most important reason is, that we try to present in this book that stuff, that is very fundamental and that is not presented at other freely accessible places in a beginner friendly fashion. And there is a lot of this which seems to be more important than games. The other reason is, that for a section about games we would have to make a lot of decisions in advance: 2D or 3D game, action game or strategy game. Using a game engine, or only a simple libraray like cairo, sdl2, raylib or gogot? When using a game engine, then the decision which one we should use, and which Nim bindings set is not a simple decision, and we should try to ensure that the bindings are actively developed and so should work in a few years with Nim 2.0 still. And finally there are already some nice tutorials for game programming available, see for example

Actually game programming is not that difficult, when we have a nice library with a good tutorial available. Game programming can be much fun, which is great, but actually we do not learn that much when we move some sprites over the screen. On the other hand, advanced game programming, by using a big library like Godot, doing all with basic libs like SDL2 or Raylib, or developing your own game engine based on OpenGL or Vulkan, is a very demanding task.

So maybe we will add a section about game programming at the end, when the rest of the book is done, or maybe when the next edition of the book is published.

nimble install packageName

nimble install https://github.com/user/packageName

import npeg

let p = peg("str"):
  str <- +{'0'..'9'}

echo p.match("123").ok

import npeg

let p2 = peg("term"):
  term <- dig * *(op * dig)
  dig <- +{'0'..'9'}
  op <- {'+', '-'}

echo p2.match("1+23").ok

The npeg module offers plain string captures and more flexible code block captures.

String Captures

import npeg

let p = peg("line"):
  line <- +(space * >word)
  word <- +{'a'..'z'}
  space <- *' '

let m = p.match(" one   two three    ")
if m.ok:
  echo m.matchLen
  echo m.captures

16
@["one", "two", "three"]

Code Block Captures

import npeg, tables

type T = Table[string, string]

let p = peg("command", t: T):
  command <- >com * '(' * >pos * ',' * >pos * ')':
    # echo $1, $2, $3
    t["action"] = $1
    t["x"] = $2
    t["y"] = $3
  com <- "moveTo" | "lineTo"
  pos <- +{'0'..'9'}


var input: T
if p.match("moveTo(12,20)", input).ok:
  echo input["action"], ": ", input["x"], ", ", input["y"]

Simple patterns

let parser = patt *(*' ' * > +(1-' '))
echo parser.match("   one two three ").captures

"Look ahead" operators

In computer science a macro (short for "macro instruction") is a rule or pattern that specifies how a certain input should be mapped to a replacement output. Meta-programming is a programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can read, generate, analyze or transform other programs, and even modify itself while running.

Legacy programming languages like C or assembly languages support already some form of macros, which generally work directly on the textual representation of the source code.

A common use of textual macros in assembly languages was to group sequences of instructions, like reading data from a file or from the keyboard, to make that operations easily accessible. The C programming language uses the #define pre-processor directive to introduce textual macros. Macros in C are generally single line text substitutions which are processed by a pre-processor program before the actual compiling process. Some examples of common C macros are

The basic C macro syntax is that the first continues character sequence after the #define directive is replaced by the C pre-processor with the rest of that line. The #define directive has some basic parameter support which was used for the sqr() macro above. C macros have the purpose to support named constants and to support simple parameterized expressions like the sqr() from above, avoiding the need to create actual functions. The C pre-processor would substitute each occurrence of the string PI in the C source file with the float literal 3.1415 and the term sqr(1+2) with (1+2)*(1+2).

A Nim macro is a function that is executed at compile-time and transforms a Nim syntax tree into a different tree. This can be used to add custom language features and implement domain-specific languages (DSL). While macros enable advanced compile-time code transformations, they cannot change Nim’s syntax.

The macro keyword is used similar to proc, func and template to define a parameterized code block which is executed at compile time and consists of ordinary Nim code and meta-programming instructions. The meta-programming instructions are imported from the macros module and are used to construct an Abstract Syntax Tree (AST) of the Nim language. This AST is created by the macro body at compile time and is returned by the macro as untyped data type. The parameter list of macros accept ordinary (static) Nim data types and additional the data types typed and untyped, which we already used for templates. We will explain the differences of the various possible data types for macro parameters later in more detail, after we have given a first simple macro example. Note that Nim macros are hygienic by default, that is symbols defined inside the macro body are local and do not pollute the name space of the environment. As macros are executed at compile time, the use of macros may increase the compile time, but their use does not impact the performance of the final executable — in some cases the use of clever macros may even improve the performance of our final program.

To verify that macros are really executed at compile time, we will start with a tiny macro that contains only an echo statement in its body:

When we compile above code the compiler prints the message "Macro argument", as it processes the macro body. When we run the program, we get only the output "calling macro m1" from the main() proc, as the macro m1() does return an empty AST only. The carefully reader may wonder, why the echo() statement in the macro body above works at all, as the parameter of macro m1() is specified as ordinary string, not as static[string]. So the type of s in the macro body should be a NimNode. Well, maybe an echo() overload exists that can work with NimNodes, or maybe, as we pass a string constant to macro m1(), in this concrete case s is indeed an ordinary string in the macro body. Maybe we should have used s: static[string] as parameter type, which would give us the exact same results.

As macros are executed at compile time, we can not really pass runtime variables to it. When we try, we would expect a compiler error:

But with the current compiler version 1.5.1 that code compiles and prints the message "str", which is a bit surprising. To fix this, we can change the parameter type to static[string], which guarantees that we can indeed pass only compile time constants. Our last example would give a compile error in this case, while the one before with the string constant would work as expected.

Now let us create macros which actually creates an AST which is returned by the macro and executed when we run our program. For creating an AST in the macro body we have various options: we can use the parseStmt() function or the "quote do:" notation to generate the AST from regular program code in text form, or we can create the syntax tree directly with expressions provided by the macros module, e.g. by calls like newTree() or newLit() and such. The later gives us the best control over the AST generation process, but is not easy for beginners. The good news is that Nim now provides a set of helper functions like dumpTree() or dumpAstGen() which shows us the AST representation of a Nim source code block as well as the commands which we can use to create that AST. This makes it for beginners much easier to learn the basic instructions necessary to create valid syntax trees and to create useful macros.

We will start with the simple parseStmt() function which generates the syntax tree from the source code text string that we pass it as argument. This seems to be very restricted, and maybe even useless, as we can write the source code just as ordinary program text outside of the macro body. That is true, but we can construct the text string argument that we pass to the parseStmt() function with regular Nim code at compile time. That is similar as having one program, which generates a new source code string, saves that string to disk, and finally compiles and runs that created program. Let us check with a fully static string that parseStmt() actually works:

When we compile and run above program, we get the output "We like Nim". The macro m1() is called at compile time with the static parameter str and returns an AST which represents the passed program code fragment. That AST is inserted into our program at the location of the macro call, and when we run our program the compiled AST is executed and produces the output.

Of course executing a fully static string this way is useless, as we could have used regular program code instead. Now let us investigate how we can construct some program code at compile time. Let us assume that we have an object with multiple fields, and we want to print the field contents. A sequence of echo() statements would do that for us, or we may use only one echo() statement, when we separate the field arguments each by "\n". The with module may further simplify our task. But as we have to print multiple fields, not an array or a seq, we can not directly iterate over the values to process them. Let us see how a simple text string based macro can solve the task:

In this example we pass the name of our object instance as static string to the macro, while we pass the fields not as string, but as list of untyped values. The passed static string is indeed an ordinary Nim string inside the macro, we can apply sting operations on it. But the field names passed as untyped parameters appear as so called NimNodes inside the macro. We can use the repr() function to convert the NimNodes to ordinary strings, so that we can use string operations on them. We iterate with a for loop over all the passed field names, and generate echo() statements from the object instance name and the field names, each separated by a newline character. Then all the statements are collected in a multi-line string s and are finally converted to the final AST by the parseStmt() function. In the macro body we use the echo() statement to verify the content of that string. As the macro is executed during compile time, we get this output when we compile our program:

And when we run it we get:

Well, not a really great result for this concrete use case: We have replaced three echo() commands with a five lines macro. But at least you got a feeling what macros can do for use.

As Nim is a statically typed programming language, all variables and proc parameters have a well defined data type. There is some form of exception from this rule for OR-types, object variants and object references: OR-types are indeed no real exception, as whenever we use an OR-type as the type of a proc parameter, multiple instances of the proc with different parameter types are created when necessary. That is very similar to generic procs. object variants and object references built indeed some form of exception, as instances of these types can have different runtime types that we can query with the case or with the of keyword at runtime. Note that object variants and references (the managed pointers itself, not the actual data allocated on the heap) occupy always the same amount of RAM, independent of the actual runtime type. (That is why we can store object variants with different content or references to objects of different runtime types using inheritance in arrays and sequences.)

For the C sqr() macro from the beginning of this section, there is no real restriction for the argument data types. The sqr() C macro would work for all numeric types that support the multiply operation, from char data type over various int types to float, double and long double. This behaviour is not really surprising, as C macros are only a text substitution — by the * multiply operator for our sqr() macro. Actually the C pre-processor would even accept all data types and even undefined symbols for its substitution process. But then the C compiler would complain later.

Nim macros and Nim templates do also some form of code substitution, so it is not really surprising that they accept not only well defined data types, but also the relaxed types typed and untyped.

As parameters for Nim’s macros we can use ordinary Nim data types like int or string, compile time constants denoted with the static keyword like static[int], or the typed and untyped data types. When we call macros then the data types of the parameters are used in the same way for overload resolution as it is done for procs and templates. For example, if a macro defined as foo(arg: int) is called as foo(x), then x has to be of a type compatible to int.

What may be surprising at first is, that inside the macro body all parameter types have not the data type of the actual argument that we have passed to the macro, but the special macro data type NimNode which is defined in the macros module. The predefined result variable of the macro has the type NimNode as well. The only exception are macro parameters which are explicitly marked with the static keyword to be compile time constants like static[string], these parameters are not NimNodes in the macro body, but have their ordinary data types in the macro body. Variables that we define inside the macro body have exactly that type that we give to then, e.g. when we define a variable as s: string then this is an ordinary Nim string variable, for which we can use the common string operations. But of course we have always to remember that macros are executed at compile time, and so the operations on variables defined in the macro body occur at compile time, which may restrict a few operations. Currently macros are evaluated at compile time by the Nim compiler in the NimVM (Vitual Machine) and so share all the limitations of the NimVM: Macros have to be implemented in pure Nim code and can currently not call C functions except those that are built in the compiler.

In the Nim macros tutorial the static, typed and untyped macro parameters are described in some detail. We will follow that description, as it is more detailed as the current description in the Nim compiler manual. As these descriptions are very abstract, we will give some simple examples later.

Static Macro Parameters

Untyped Macro Parameters

Typed Macro Parameters

Code Blocks as Arguments

echo("1 + 2 = "):
  1 + 2

import macros

macro m1(x: static[int]): untyped =
  echo "executing macro body"

m1(3)

import macros

macro m1(x: int): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

import macros

macro m1(x: untyped): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

In the section before we learned about the parseStmt() function which is used in a macro body to compile Nim code represented as a multi-line string to an abstract syntax tree representation. Macros uses as return type the "untyped" data type, which is compatible with the NimNode type returned by the parseStmt() function.

The quote() function and the quote do: construct has some similarity with the parseStmt() function: It accepts an expression or a block of Nim code as argument and compiles that Nim code to an abstract syntax tree representation. The advantage of quote() is that the passed Nim code can contain NimNode expressions from the surrounding scope. The NimNode expressions have to be quoted using backticks.

As a first very simple example for the use of the quote do: construct we will present a way to print some debugging output.

Assume we have a larger Nim program which works not in the way that we expected, so we would add some echo() statements like

Instead of the echo() statement we would like to just write show(currentSpeed) to get exactly the same output. For that we need access not only to the actual value of a variable, but also to its name. Nim macros can give us this information, and by using the quote do: construct it is very easy to create our desired showMe() macro:

When we compile and run that code we get:

In the macro body we use the proc toStrLit() from the macros module which is described with this comment: "Converts the AST n to the concrete Nim code and wraps that in a string literal node" So our local variable n in the macro body is a NimNode that now contains the string representation of the macro argument x. We use the NimNode n enclosed with backtics in the quote do: construct. It seems that writing this macro was indeed not that difficult, but actually it was only that easy because we have basically copied the dump() macro from the sugar module of Nim’s standard library.

Let us investigate our show() macro in some more detail to learn more about the inner working of Nim macros. First recall that macros always have a return value of data type untyped, which is actually a NimNode. The quote do: construct gives us a result which we can use as return value of our macro. Sometimes we may see macros with no result type at all, which is currently identical to the untyped result type. As the macro body is executed at compile time, the quote do: construct is executed at compile time as well, that is that the code block which we pass to the quote do: construct is processed at compile time and the quoted NimNodes in the block are interpolated at compile time. For our program from above the actual echo() statement in the block is then finally executed at program runtime. To prove how this final echo() statement looks we may add as last line of our macro the statement "echo result.repr" and we would then get the string "echo "a * sqrt(b)", ": ", a * sqrt(b)" when we compile our program again.

In the two sections before we used the functions parseStmt() and quote() to build the AST from a textual representation of Nim code. That can be convenient, but is not very flexible. In this section we will learn how we can build a valid AST from scratch by calling functions of the macros module. That is not that easy, but this way we have the full power of the Nim meta-programming available.

Luckily the macros module provides some macros like dumpTree() and dumpAstGen() which can help us get started. We will create again a macro similar to the show() macro that we created before with the quote do: construct, but now with elementary instructions from the macros module. This may look a bit boring, but this plain example is already complicated enough for the beginning, and it shows use the basics to construct much more powerful macros later.

The core code of our debug() macro would look in textual representation like

That is for debugging we would like to print an expression first in its string representation, and divided by a colon the evaluated expression. The dumpTree() macro can show us how the Nim syntax tree for such a print debug statement should look:

When we compile this code we get as output:

So the Nim syntax tree for the echo() statement from above is a statement list consisting of an echo() command with two string literal arguments and a last argument which is built with the infix + operator and the two arguments a and b. So we can see how the AST that we would have to construct would have to look, but we still have no idea how we could construct such an AST in detail. Well, the macros module would contain the functions what we need for that, but it is not easy to find the right functions there. The dumpAstGen() macro can list us exactly the needed functions:

This is a nested construct. The most outer instruction constructs a new tree of Nim Nodes with the node type statement list. The next construct creates a tree with node kind command, which again contains the ident node with name echo, which again contains two literals and the infix + operator.

Indeed we can use the output of the dumpAstGen() macro directly to create a working Nim program:

When we compile and run that code, we get the output:

So the AST from above is fully equivalent to the one line echo() statement. But now we would have to investigate how we can pass an actual expression to our macro and how we can use that passed argument in the macro body — first print its textual form, and then the evaluated value separated by a colon. And there is one more problem: That nested macro body from above is not really useful for our final dump() macro, as we would like to be able to construct the NimNode, that is returned by the dump() macro step wise: Add the echo() command, then the passed expression in string form, and finally the evaluated expression. So let us first rewrite above macro in a form where the AST is constructed step by step. That may look difficult, but when we know that we can call the newTree() function with only one node kind parameter to create a empty tree of that kind, and that we can later use the overloaded add() proc to add new nodes to that tree, then it is easy to guess how we can construct the macro body:

First we create the tree empty three structures of node kinds statement list, command and infix operator. Then we use the overloaded add() proc to populate the threes, using procs like newIdentNode() or newLit() to create the nodes of matching types as before. When we run our program with the modified macro version m2() we get again the same output:

The next step to create our actual dump() macro is again easy — we pass the expression to dump() as an untyped macro parameter to the macro, convert it to a NimNode of string type and use that instead of the newLit("a + b") from above. In our second macro, where we used the quote do: construct, we applied already toStrLit() on an untyped macro parameter, so we should be able to reuse that to get the string NimNode. Instead we would have to apply the stringify operator additional on that value. But a simpler way is to just apply repr() on the untyped macro argument to get a NimNode of string type. And finally, to get the value of the evaluated expression in our dump() macro, we add() the untyped macro parameter directly in the command three — that value is evaluated when we run the macro generated code.

Again, we get the desired output:

So our dump() macro called still m2() is complete and can be used to debug arbitrary expression. Note that this macro works for arbitrary expressions, not only for numerical ones. We may use it like

Now let us extend our debug() macro so that it can accept multiple arguments. The needed modifications are tiny, we just pass instead of a single untyped argument an argument of type varargs[untyped] to the debug macro, and iterate in the macro body with a for loop over the varargs argument:

When we compile and run that code we get:

As one more simple example we will show how we can create our own assert() macro. The assert() has only one argument, which is a expression with a boolean result. If the expression evaluates to true at program runtime, then the assert() macro should do nothing. But when the expression evaluates to false, then this indicates a serious error and the macro shall print the expression which evaluated to false, and then terminate the program execution. This is basically what the assert() macro in the Nim standard library already does, and the official Nim macros tutorial contains such an assert() macro as well.

Arguments for our assert() macro may look like "x == 1 +2", containing one infix operator and one left-hand and one right-hand operand. We will show how we can use subscript [] operators on the NimNode argument to access each operand.

As a first step, we use the treeRepr() function from the macros module to show us the Nim tree structure of a boolean expression with an infix operator:

When we compile that program, then the output of the treeRepr() function shows us, that we have passed as argument an infix operator with two operands at index position 1 and 2.

Now let us create an assert() macro which accept such a boolean expression with an infix operator and two operands:

The first two function calls expectKind() and expectLen() verify that the macro argument is indeed an infix operator with two operands, that is the total length of the argument is 3. The symbol nnkInfix is an enum value of the NimNodeKind data type defined in the macros module — that module follows the convention to prepend enum values with a prefix, which is nnk for NimNodeType in this case. In the macro body we use the subscript operator [0] to access the operator, and then apply repr() on it to get its string representation. Further we use the subscript operators [1] and [2] to extract the two operands from the macro argument and store the result each in a NimNode lhs and rhs. Finally we create the quote do: construct with its indented multi-line string argument and the interpolated NimNode values enclosed in backtics. The block after the quote do: construct checks if the passed arg macro argument evaluates to false at runtime, and raises an exception in that case displaying the reconstructed argument.

We have to admit that this macro is not really useful in real life, as it is restricted to simple boolean expressions with a single infix operator. And what it does in its body makes not much sense: The original macro argument is split in tree parts, the infix operator and the two operands, which are then just joined again to show the exception message. But at least we have learned how we can access the various parts of a macro argument by use of subscript operators, how we can use the treeRepr() function from the macros module to inspect a macros argument, and how we can ensure that the macro argument has the right shape for our actual macro by applying functions like expectKind() and expectLen() early in the macro body.

All macros and templates can also be used as pragmas. They can be attached to routines (procs, iterators, etc), type names, or type expressions. In this section we will show a small example how a proc pragma can be used to print the proc name whenever a proc annotated with that pragma is called:

We start with the dumpAstGen() macro applied on a test() proc which contains an echo() statement. So when we compile that code, we get an initial idea how a NimNode that shall print the proc name should look. To use pragma macros, we annotate the proc with the macro name enclosed in the pragma symbols {..}. The annotated proc is then passed to the pragma with that name in form of a syntax tree. Our goal is to to add a NimNode to this tree that prints the proc name of the passed AST. To do that we have to know two important points: For the proc that is passed as untyped data type to our macro, we can use the function body() to get the AST representation of the body of the passed proc, and we can use name() to get the name of that proc. The functions body() and name() are provided by the macros module of Nim’s standard library. In our macro pm() we first verify, that the passed argument is really of node kind ProcDef. Then we create a new NimNode, which calls the echo() function with the proc name as parameter. And we insert that node at position 0 into the body of the passed proc. Finally we return the modified AST.

When we run our program, we get this output in the terminal window:

Let us assume we have an object type which has some fields which are all sequences with the same base type, and we need an iterator to iterate over all the container elements. Indeed this may happen when the different seqs contain subclasses of the same parent class as in

Maybe we do not want to write all the for loops in the iterator body manually. One solution is to create a pragma macro, which creates the for loops in the iterator body for us:

We start again with a dumpAstGen() call which shows us the shape of the for loop node. In that node we only have to replace two newIdentNode() calls so that the fields names can be provided by iterating over an array of strings, and the object name is taken from the iterator parameter. To get the object name, we first use o.treeRepr to see the whole parameter structure, and then params.treeRepr to get the structure of the parameters passed to our iterator. Using subscript operators we get the actual object name. We insert each new node that we create in the for loop with a call of insert(body(o), body(o).len, node) as new last node in the body of the iterator. We can create a more flexible variant of our above macro, when we pass the actual field names as additional parameter to the pragma macro:

When we run this macro or the one before, we get

Earlier in the book we have already learned how we can define new procs and templates which can be used as operators. In this section we will learn how we can create a macro that does not only create an operator that can work on existing variables, but that can be used to create new variables. In Nim we use the var or let keyword to create new variables. Some other languages allow to create new variables on the fly by using just "=", ":=" or "!=" to create new variables.

Again dumpAstGen() shows us the structure of the needed AST. We use repr() to get the string representation of the two macro arguments and replace in the dumpAstGen() output the arguments of the newIdentNode() calls with that values. When we compile and run above program we get

For the case that we should really intend to use such a macro in our own code, we should of course add some code to the macro to check that the passed arguments have the correct content.

References:

Creating new threads is always some overhead, so it can make sense to create a pool of threads, which we then can use to executes parts of our program.

Using spawn to execute a proc by one thread of the pool

import std/threadpool
proc sum(i: int): int =
  var j = 0
  while j < i:
    inc(j)
    result += j

proc main =
  var a: FlowVar[int] = spawn sum(1e7.int)
  var b = spawn sum(1e7.int)
  echo ^a , " ", ^b

main()

import std/threadpool
from std/os import sleep
proc doSomeWork =
  echo "not really working that hard..."
  sleep(1000) # sleep 1000 ms

proc main =
  var userInput: FlowVar[string] = spawn readLine(stdin)
  while not userInput.isReady:
    doSomeWork()
  echo "You finally entered: ", ^userInput

main()

The parallel statement

import std/threadpool
{.experimental: "parallel".}

proc sum(i, j: int; a: array[8, int]): int =
  for k in i .. j:
    result += a[k]

proc main =
  var a: array[8, int]
  for i in a.low .. a.high:
    a[i] = i

  var s1, s2: int
  parallel:
    s1 = spawn sum(0, 3, a)
    s2 = spawn sum(4, 7, a)
  echo s1, " + ", s2

main()

When for some reason we can not use the threadpool module, or we need more control over the various threads, then we can create our own treads:

The createThread() procedure is provided by the threads module, which is part of the system module — for that reason we do not have to explicitly import it. The proc that we want to execute in its own, newly created thread has to be annotated with the {.thread.} pragma and has to use one single parameter. We pass the generic Thread[T] variable, the proc to execute and the proc parameter to createThread(). The Thread variable must have the same generic type as the parameter of the proc that we want to execute. In our example that parameter type is a plain integer, but of course we can use other data types including objects, tuples or container types like sequences.

As createThread() does not return a result, we call echo() in our sum() proc to show what is going on. Actually calling echo() from within a proc running as a thread may be not a good idea, as multiple echo() calls from different threads may interfere. We may use the locks module to make the output operation atomic, but to keep our example short and simple we ignore that problem for now. The code above creates two newly created threads, which in our case run the same proc with the same data. If there is more than one CPU core available, then the two threads should be executed in parallel by the OS. After launching our new treads, we can use the joinThreads() procedure to wait for the termination of all treads — we should generally do that before our app terminates itself.

When we use the threadpool and spawn() to execute a function by one of the threads of the pool, we get immediately the result of the executed function back when the work of the function is done.

Threads created with the createThread() function of the threads module do not directly return a result but may be executed for a long time period, often for the whole lifetime of the main process. Generally it is necessary to exchange messages and data between these types of threads — among multiple child threads themselves or among them and the process’s main thread. For this message- and data-exchange Channels can be used. Nim’s Channels use internally a queue for sending data from one thread to another thread. A queue is a first-in-first-out (FIFO) data structure — items put in first will also be extracted first. That way the receiving thread will receive the items in the same order as the sending thread has sent them.

The generic Channel[T] data type and the functions to use it are provided by the system module, so we do not need to import them. Channels should be used only for Threads of the threads module, but not for the hidden threads of the threadpool module. Channels allow to send messages and data only in one direction, for bidirectional communication we would need two separate channels. Variables of the Channel data type are generally defined at the global scope, to avoid problems with the thread local garbage collector, and the generic type of the Channels determines the data type of the messages that we can send through the Channel. The sent data is deeply copied by the Channel, which may be not that efficient for large data packages.

In the code below we will present a very simple example for the use of one single Channel. The proc sum() sums up again the first n natural integer numbers, but this time the function sums up the numbers in chunks, and send the sum of each chunk over the Channel to the parent thread:

The proc sum() sums up continuously 4 more numbers, and then sends the partial sum into the channel. The generic Channel[int] variable ch is defined in global scope. In the main() proc we create the child thread, open the Channel and read the Channel data with calls to recv() until we get a zero value as terminating condition. Finally we call joinThreads() to ensure that the child thread was really terminated and call close() on the channel to close it. Note that in sum() we use an additional send() call to send the last partial sum which may have less than 4 sumands and so may not have been sent. Instead of this additional send() call in the while loop a condition like if j mod 4 == 0 or i == j: could be used of course. When we are done we send the zero value to indicate to the parent thread that we are done. This way the parent thread will not wait for more data that never got send. In the main() proc we use recv() to read the data from the Channel. Recv() would block if data is not yet available. Instead we could use tryRecv() with returns a tuple, with the field dataAvailable indicating if there is already something to read available. The open() function accepts as second optional argument the number of items that can be buffered in the internal items queue of the channel. If that limit is reached, further calls to send() would block until the reading thread has read the next item. If we restrict the maximum number of items in the Channel, we may use instead of send() which may block when the channel is full, trySend() which just returns false for this case without blocking.

Of course the code example from above makes not much sense, as there is no real useful work done in parallel, and as there is no reason for sum() to not just sum up all the elements immediately. But the example should show you the basic use of Channels, including the need for having a terminating condition.

A race condition may occurs when two or more threads attempt to read and write to a shared resource at the same time. Such behavior can result in data corruption or unpredictable results that are difficult to debug. Let us consider this tiny example, where two threads increase the value of a global integer variable:

In the code above the two threads are running concurrent, and in parallel when your CPU has at least two physical cores. Each thread increases the global counter variable N times, so one may expect a final result of 2 * N. But at least when the threads are running in parallel the actual result will be a random value between N and 2 *N. The problem is, that the threads do not increase the global counter in one atomic step, but create a local copy, increase the value of the copy and write the value back. When the other thread had modified the global counter variable in between, that modification is overwritten. When the two threads would run not in parallel but concurrent on only one CPU core, then the actual result may depend on the way how the OS does the actual task switching.

These kind of problems are sometimes called race conditions, because the actual behaviour is determined by the order in which the various threads access the data. In the example code the actual problem results from the copying into the local variable and later copying the value back — a plain inc() executed on the global variable may work. We used the local copy here to make the problem visible. Whenever we would work in such an unordered way onto more complicated data like strings or objects, we would get corrupted data. This example should raise your awareness to all the problems which may occur when multiple threads access global data in an uncontrolled way.

We have already learned about Channels, which provide a way to exchange data between threads without the use of global variables. Other methods to protect global variables from uncontrolled access which can lead to corrupted states are locks, mutexes or semaphores. We will give an example to due access control by use of Locks in the next section. In the example above we used as global data a primitive value data type. Even more problems may occur, when we try to use global references data types: In the past the Nim standard library provided special functions like allocShared() to allocate pointer and reference data types that can be accessed from multiple treads. But as Nim’s thread handling may change and improve in the future further, we will not try to discuss all these details here. It should be enough that you have a feeling for the problems that my arise from executing multiple threads with shared data — for the details you should consult the documentation of Nim’s standard library and the compiler manual.

While Nim’s Lock data type and the corresponding functions are defined in the locks module of the standard library, that module contains only minimal explanations, so we have to consult the experimental section of the compiler manual.

In computer science, a lock or mutex (from mutual exclusion) is a synchronization primitive that enforces limits on access to a resource when there are many threads of execution. Before that resource is accessed, the lock is acquired, and after the resource is accessed, it’s released. The simplest type of lock is a binary semaphore. It provides exclusive access to the locked data. Following this definition from Wikipedia, Nim’s locks seems to be actually binary semaphores.

Nim’s Locks are generally used together with the guard pragma, which we can attach to a global variable that is accessed from more than one thread. With the guard pragma attached, each thread has to first acquire the lock before it is allowed to access that variable. If the lock is already acquired by another thread, acquire blocks until that other thread releases the lock to indicate that it is done with its access. Of course these possible blocking can decrease the total performance, so each thread should acquire the lock only when it needs really access to the protected data and release the lock as soon as possible.

We can use the template withLock to access the guarded global variable in a block — withLock() acquires the given lock, and releases the lock again at the end of the block. Accessing a guarded variable outside a withLock() block would give a compile time error.

When you now run the above code, counter should always have the desired value 2 * N. Note that replacing the withLock with a plain acquire() and release() pair seems not to work for locks that are used as guards — but actually there is no reason to do that, the withLock block is easier to use and ensures that acquire() and release() is always used in matching pairs.

Whenever a proc that is running as its own thread is raising an uncaught exception, then the whole process is terminated and a stack trace with the corresponding error message is displayed in the terminal window. This applies not only to the threads of the threads module, but also when the spawn() function is applied to run functions by one of the threads in the pool.

References

For IO-bound task the use of async/await can actually have performance benefits.

The multi-threaded program execution, that we described in the previous sections, is some form of preemptive multitasking, where switching between the active threads occurs at arbitrary time intervals controlled by the OS. But the async/await pattern is a form of co-operative multitasking, which provides the user with full control of the code execution. We can pause the code execution by using the await keyword when it really makes sense. e.g. when we have to wait for new data packets or events, and immediately enable execution of a different code path.

As for this form of co-operative multitasking only the code execution path is changed, but no switching between threads is necessary, additional overhead can be avoided, and typical problems of multi-threading, like the passing of data between different threads or race conditions do no exist.

So at least in theory the co-operative multitasking controlled by the async/await pattern is more efficient, and for maximum performance it can be combined with threading and parallel program execution.

The core elements of Nim’s async/await framework are provided by the modules std/asyncdispatch and std/asyncfutures.

These modules provide a dispatcher, a generic Future[T] type implementation, and the async macro which allows asynchronous code to be written in a synchronous style with the await keyword.

The asyncdispatch module implements a global dispatcher (technically one per thread) that is responsible for running the procedures that are registered with it.[51]

Build on top of these two modules there exists various modules for asynchronous communication: Module std/asyncnet implements a high-level asynchronous sockets API and std/asynchttpserver implements a high performance asynchronous HTTP server. Some other modules like std/httpclient support synchronous and asynchronous data transfers.

Nim’s async/await framework is not part of the language itself, but implemented with macros and meta-programming and use of Nim’s iterators. The underlying implementation is based on epoll on Linux, IO Completion Ports on Windows and select on other operating systems.

Currently Nim’s async/await uses only one single thread on its own, but applications can combine it with multiple parallel running threads. As an alternative implementation we could use https://github.com/status-im/nim-chronos, which provides similar functionality.

Asynchronous procedures are marked by the {.async.} pragma and must return a generic Future[T] type or return no result at all. In the later case a Future[void] is assumed. A Future, also called Promise in other languages, is a generic container type which holds a value which is not yet available, but which may be available in the future. So a Future has some similarity with the generic FlowVar type that we used as return types for threads of Nim’s threadpool.

Inside asynchronous procedures the keyword await can be used to call other asynchronous procedures or procedures which return a Future type.

The await keyword will suspend the code execution until the awaited Future completes. After completion the asynchronous procedure will resume its execution. During the period when an asynchronous procedure is suspended other asynchronous procedures will be run by the dispatcher.

The generic Future[T] data type

import std/asyncdispatch

proc cb(f: Future[string]) =
  echo "executing callback: ", f.read

let f1: Future[string] = newFuture[string]()
echo f1.finished

f1.callback = cb

f1.complete("Nim and its future")
#f1.fail(newException(ValueError, "Future failed"))

We will start our explanations with a very simple asynchronous program, which will not do an actual asynchronous data transfer yet, but an asynchronous sleep (wait). The asynchronous sleep called sleepAsync() actually behaves very similar to the asynchronous data transfer functions, that is the execution of the actual code path is suspended until a hardware condition is fulfilled, and the dispatcher continues with the code execution.

In the code example above we import the asyncdispatch module and have attached the {.async.} pragma to our tick() procedure. As the tick() proc does not return any actual data, we use Future[void] as return type — actually we could leave out the return type for this case. We call tick() two times with different string arguments and use the function epochTime() to measure the total execution time of our program. When we compile and run the code we get this output:

The result is not really surprising, as for each call of proc tick() the loop in its body is executed two times, generating a 100 ms delay for each iteration. But the output will drastically change when we call instead of the ordinary sleep() function the function sleepAsync() provided by the asyncdispatch module:

The two calls of the tick() proc each returns nearly instantly a Future[void] object, no waiting happens here. The use of the await keyword in the proc body causes the proc to suspend its execution and control flow returns to the calls site immediately. But at the same time the asynchronous tick() proc got registered by the dispactcher loop, so that it can resume its execution.

The returned Future object encapsulates the actual return type of the call — in this case only void — and gives us a reference that we can use to ask the dispatcher whether our call has completed or not.

But futures can’t get resolved by themselves, we need to actually run the dispatcher in order for any of the code registered with it to resume its execution. Remember all of this is still running in a single thread of execution. There are many ways to run the dispatcher, but in this case it is done by the waitFor call. When we run waitFor, the dispatcher will run in a loop until the given future is completed, and the proc which has returned that future is removed from the dispatcher loop. WaitFor actually calls poll() in a loop until the future is finished, and then returns the generic value of the future — in the code above waitFor returns no actual result, as we used a Future of void type.

We can use the operators and or or to combine multiple futures, in this way we can wait until all of them or at least one of them completes. Note that the dispatcher loops stops when waitFor() succeeds, so when we wait only for one future and that one finished, then the dispatcher loop stops and other futures may stay unfinished.

We can use the function finished() to check if a future variable is already finished. When a future is finished it means that either the value that it holds is now available or it holds an error instead. The latter situation occurs when the operation to complete a future fails with an exception. We can distinguish between the two situations with the failed() function. Future objects can also store a callback procedure which will be called automatically once the future completes, see the example in the previous section.

In our example code above we called waitFor f1 — this is necessary to actually execute the dispatcher loop, and to wait for the future f1 to complete. We could have used waitFor f1 and f2, or waitFor f1 or f2 to wait for completion of both futures or one of them. The result would be identical in this case, as the proc that returns f1 and f2 is identical and returns always after 2 loop iterations.

The important result of this modified code is, that the proc execution alternates, and the total runtime of the program is only 0.2 ms. The reason for this is, as we already explained, that the use of the await keyword in our tick() proc suspends the execution, and so immediately the next call of tick() with "BBB" as argument is executed.

As one more simple example let us investigate this code, where two different asynchronous procs are executed:

As both asynchronous procs use different arguments when they call sleepAsync(), they are not executed strictly alternating, so the numbers 2 and 3 are printed with no letter in between:

In this example we do not call waitFor() directly on our actual asynchronous procs, but on sleepAsync() from asyncdispatch. As the procs numbers() and letters() got registered by the dispatcher, they are executed by the dispatcher loop, but only as long as determined by waitFor sleepAsync(1500). So the execution of the dispatcher loops stops already before letters() is really done, and letters d and e got never printed. The fact that the printed time values can be a few ms larger than the actual specified sleep times should not surprise us, as some more code is executed in our procs, and as the dispatcher loop itself may need some tiny execution time. When an exact timing should be required, we may use the std/times module to read the exact time and adjust the actual delays. Also note that async/await as a co-operative approach of multitasking also means that long running tasks can delay the execution of other tasks unexpectedly: Imaging that in our code above the numbers proc would contain a lot of additional code that takes more than 250 ms to run — that would confuse the whole timing scheme. As async/await is most often not used to create actual delays, but for asynchronous network and IO operations, we will not discuss the problems of exact timing here in detail. The linked paper of P. Munch discusses this topic in some more detail and offers some possible solutions for more accurate timings.

The module std/httpclient of Nim’s standard library provides procedures for synchronous and asynchronous file transfer. Let us start with this simple synchronous example to download two small text files from an URI:

We have uploaded the two plain text files to that location in advance, and when we compile and run above code we should get:

Nim’s API documentation for std/httpclient shows us how we can do the download in an asynchronous way — at least for one single file:

In this example we use an asynchronous HTTP client, for which the overloaded proc getContent() returns a Future[string] in this case. The call of waitFor waits for the download to finish and returns the actual content of the future, which is a string containing the page content.

With the knowledge which we get from our previous example with sleepAsync(), we can easily modify above code to download two files asynchronous:

The combination f1 and f2 actually creates a new future of void type. We use two variables f1 and f2 of string type, and read the content with the read() proc when both futures are completed.

In the API documentation of the std/asyncnet module we find this example for a very basic chat server application:

The purpose of a chat server is, that multiple clients can connect to a running server, and then all messages that a client sends to the server got resend to all other connected clients. So one user can talk to all the other connected users.

A chat server has to perform two primary tasks:

• Listen for new connections from potential clients

• Listen for new messages from clients that have already connected to the server

All the messages that the server receive will need to be sent to every other client that is currently connected to it.

We have not a working client app yet, but in the case that you have the telnet program installed on you computer, you can already use that one to test this server. Telnet sends messages unencrypted , so its use is generally not recommended to send messages over the internet, but for testing purposes on the local net we may use it. If the telnet app is not installed on your computer, you may install it with the package manager of your OS — for Gentoo Linux we would run "emerge -av telnet-bsd". An alternative may be the use of the busybox app, which provides the telnet functionality as well.

If you have a telnet app available, you may open three terminal windows: On the first one you compile and run the server app — you will see no output in that window. In the two other terminals you type telnet localhost 12345 each. That should start the telnet app which connects to our running server. When you now type in some text, that text is echoed to both telnet windows:

Note that terminating the telnet app is not that simple — you may have to type CTRL ] first, then you get the telnet prompt, where you type quit to terminate the app.

Before we will try to explain the details of above server app, we should summarize a few facts about network communication. Then, at the end of this section, we will create a simple client app, which you can use instead of telnet to send messages to the server.

Data transfer over a network

import std/[asyncnet, asyncdispatch]

proc processClient(client: AsyncSocket) {.async.} =
  while true:
    let line = await client.recvLine()
    if line.len == 0: break
    echo line

proc serve() {.async.} =
  echo "start serve()"
  let server = newAsyncSocket()
  server.setSockOpt(OptReuseAddr, true)
  server.bindAddr(Port(12345))
  server.listen()
  while true:
    let client = await server.accept()
    let f1: Future[void] = processClient(client)

let f: Future[void] = serve()
echo "back at main scope"
waitFor sleepAsync(320000)

The client has to connect to the server, and then to watch for keyboard input from the user and for arrival of new messages from the server at the same time. So again we have to care to prevent blocking operations. Unfortunately reading user input in a terminal window is always blocking, and there is currently no input method available that is directly supported by Nim’s async/await framework. But we presented earlier in the book already a way to avoid the blocking behavior of the readLine() proc by use of Nim’s threadpool library. We will use that method again for the realLine() calls, and combine it with the async/await pattern for sending messages to the server and for watching for other messages from the server. Actually our client example program follows closely the client program from Mr. Picheta, the creator of Nim’s async/await framework, which he sketched in the Manning book years ago:

The structure of this client implementation is a bit different from the server one. The main reason is, that we have to use Nim’s threadpool and spawn to avoid the blocking behaviour of the readLine() proc. Note that our main() proc is not marked with the [.var]{.async.} pragma and contains no await statement. Only the doConnect() proc, which connects to the server and then watches for messages send by the server is marked with the async pragma and awaits the new messages. The main() proc creates the new asynchronous socket and then calls the asynchronous proc doConnect(), which actually connects to the server and enters an infinite loop watching for messages from the server. When doConnect() calls await, control flow returns immediately to our main() proc. But doConnect() has become a component of the dispatcher loop, so its infinite while loop with the await statement will gain control back later. In the main() proc we then use spawn to execute readLine() on one thread of the threadpool, and enter a different infinite while loop. This loops checks if user input is available, and calls poll() to ensure that the global dispatcher loop is executed. If there is user input available, that message is send to the server, and spawn is called again waiting for the next user input.

Of course you may wonder of this client structure really makes sense. At least it seems to work. But you may be right — the use of spawn is an important component to avoid the blocking terminal input issue, and the dispatcher loop seems to do not that much contribution.

Feel free to experiment with modified client app structures yourself.

Final remarks

proc print(i: int) =
  let c =
    if i > 31 and i < 128: char(i) else: ' '
  stdout.write("  ", c, "  ")

proc main =
  echo "Visible ASCII Characters\n"
  stdout.write("     ")
  for i in 0 .. 15:
    if i < 10:
       stdout.write(" +")
    else:
      stdout.write("+")
    stdout.write(i, "  ")
  stdout.write('\n')
  var i = 0
  while i < 128:
    if i < 10:
      stdout.write("  ")
    elif i < 100:
      stdout.write(" ")
    stdout.write(i, ' ')
    for j in 0 .. 15:
      print(i + j)
    stdout.write('\n')
    inc(i, 16)

main()

type
  T = array[-5 .. 4, int]
  T2 = array[-5 .. 4, T]

var t: T2

for d in 0 .. 1:
  if d == 0:
    echo "\nResult of i div j"
  else:
    echo "\nResult of i mod j"
  for i in -5 .. 4: # row
    for j in -5 .. 4: # col
      if i == -5 and j == -5:
          t[i][j] = int.high
      elif i == -5:
        t[i][j] = j
      elif j == -5:
        t[i][j] = i
      else:
        if j == 0:
          t[i][j] = int.high
        else:
          if d == 0:
            t[i][j] = i div j
          else:
            t[i][j] = i mod j

  for i in -5 .. 4:
    for j in -5 .. 4:
      if t[i][j] >= 0:
        stdout.write(" ")
      if t[i][j] == int.high:
        stdout.write("  ")
      else:
        stdout.write(t[i][j], " ")
    echo ""